| Serial line Burstnibbler, double toggle "asynchronous" protocol [message #142329 is a reply to message #142327] |
Sat, 06 May 2006 18:40   |
Wolfgang Moser
Messages: 632
Registered: July 2003
Karma: 0
|
Senior Member |
|
|
Heya Dag,
>> What is this getting here, code bashing? ;-))
>
> Not at all, just trying to help give you better timing margins :-)
As I said. You improve my code by reducing it
by two cycles. I answer with reducing it by
another one, answered by you with squeezing
out one code byte, ... and so on. Just team
programming ;-)
>> I prefer to do the clv as soon as possible
>
> The reason why I moved the clv was to keep your original timing
> between the writes - I thought perhaps it would be problematic if the
> writes came too close. At least when reading on a C64 you have to keep
> the data readable for a few cycles, since there will be an unavoidable
> delay in the transition detection loop.
Well, ok ... the very same difficulties seem
to exist for some decent parallel port chipsets
or their exact implementations on different
motherboards.
> Isn't it possible to keep the ATN A bit out of the equation by
> toggling bit 4 in the data direction register $1803?
Ähem, I'm soooooo stupid, yes this is another
good idea for building a (hardware level)
bitmask.
> OK, so you can make it work even without relying on the data direction
> register.
As another option, just what would fit Per's
task more efficiently in the end.
> Well, two cycles is actually quite a lot, under the circumstances :-)
Under _any_ circumstances I can think of ;-)
Womo
|
|
|
|
|
|
|
|
| Re: Serial line Burstnibbler, Was: [...] 0.4.0 of opencbm [...] [message #142332 is a reply to message #142328] |
Sun, 07 May 2006 12:08 |
Pete Rittwage
Messages: 444
Registered: March 2005
Karma: 0
|
Senior Member |
|
|
Wolfgang Moser wrote:
> Me too. I hope that Per gets done the hard
> part of making a prototype of the bare
> protocol, the bitstuffing and most importantly
> the postprocessing algorithms that stuff
> together "multilayer bitstream" pieces.
>
> _Then_ Pete Rittage, Nate Lawson et al jump
> in I hope and require me "only" for some of
> the hardest parts and for testing as always.
>
>
> Or with the MMC64. I already heard some MMC64
> people asking for a Burstnibbler-alike tool,
> that does similar things as MNib. There was
> one in Pete's forum asking about, if MNib
> could be backported to the C64... Jesus.
The main challenge that I undertook in MNIB was not the data transfer
aspect, as that was already done for me. My work has been in improving
the track cycle detection, processing the GCR data, and remastering.
One problem I see right away in taking only 2 bits at a time, is how do
you know how to piece this back together on the other end into a valid
track cycle? Currently, we take 8192 bytes all in one sweep, which
allows for differing amounts of overlap in the GCR data at different
bitrates, and then I run it through a series of different algorithms
that attempt to detect this overlap and establish how the track was
"probably" appears on the disk.
You could have some sort of "bookmark" pattern you scanned for on the
track to line it up, but that will fail on many protection tracks that
are not anything like standard DOS tracks. They might be all sync, all
$0 bytes (unformatted), all one certain byte ($AC), or a simple
multi-byte signature pattern that is repeated many times. In these, you
cannot bookmark a certain byte pattern. The only way this would seem to
work is on standard DOS sectors or something very close.
You could also try tightly timing the routine, but I'm not sure how
precise you can get with the 1541 hardware. If you just went straight
around the track 4 times transferring bits the whole way, you would get
bitshifted data, etc. since the track length might not even be a
multiple of 8 bits long. :)
Anyway, if you can get a routine that transfers a track this way, I
would be happy to help however I can to figure out how to process the
data. If it works, it would open up the ability to image disks to a
wider audience.
-
Pete Rittwage
C64 Preservation Project
http://c64preservation.com
|
|
|
|
| Re: Serial line Burstnibbler, Was: [...] 0.4.0 of opencbm [...] [message #142333 is a reply to message #142328] |
Sun, 07 May 2006 14:01 |
|
|
Originally posted by: Dag Lem
Wolfgang Moser <wnhp@d81.de.invalid> writes:
>> I've been playing with some ideas for partial GCR decoding while
>> reading bytes off the disk. Here I've been using the technique you
>> describe while reading bytes in groups of five.
>
> I gave up on this years ago. Only by thinking
> about it I could never find a suitable idea that
> not incorporates any hardware tricks.
I think I'm onto something, but the code is not pretty :-)
>> 1541. I hope you or someone else with intimate knowledge about 1541 to
>
> As long as hardware is incorporated, timing,
> cyclecounting and such, maybe. But I'm not good
> on the 1541's software side. Nearly no knowledge
> about how the DOS-ROM works, the Jobcode
> interpreter, IRQ routine and these things.
Well, if Per comes up with proof of concept code for transfer to the
C64, I guess you're in business ;-)
>> If it should prove impossible to make it work with an XA1541 cable (I
>> hope not!), there's still the option of transferring via an RR-Net
>> equipped C64.
>
> Or with the MMC64. I already heard some MMC64
> people asking for a Burstnibbler-alike tool,
> that does similar things as MNib. There was
> one in Pete's forum asking about, if MNib
> could be backported to the C64... Jesus.
Perhaps it's not as bad as it sounds :-) You could use CC65 to port C
code, if that's what you're using. An advantage of transferring to the
C64 is that you wouldn't have to worry about any transfer related
timing problems.
--
Dag Lem <resid@nimrod.no>
|
|
|
|
| Re: Serial line Burstnibbler, Was: [...] 0.4.0 of opencbm [...] [message #142334 is a reply to message #142332] |
Sun, 07 May 2006 14:23 |
|
|
Originally posted by: Dag Lem
Pete Rittwage <rcade@rittwage.com> writes:
> One problem I see right away in taking only 2 bits at a time, is how
> do you know how to piece this back together on the other end into a
> valid track cycle? Currently, we take 8192 bytes all in one sweep,
> which allows for differing amounts of overlap in the GCR data at
> different bitrates, and then I run it through a series of different
> algorithms that attempt to detect this overlap and establish how the
> track was "probably" appears on the disk.
I think the crux of Per's idea is to do overlapped full track reads,
like this:
1: transfer bits 0, 1
2: transfer bits 1, 2
3: transfer bits 2, 3
4: transfer bits 3, 4
5: transfer bits 4, 5
6: transfer bits 5, 6
7: transfer bits 6, 7
He can then overlay and (bit)shift the data until it all lines up to
the same 8192 bytes you'd read in one go using a parallel cable.
Note that in general it isn't enough to simply shift bytes, if there
are no sync marks you're not guaranteed to get byte aligned reads. I
don't know whether that's anything to worry about in practice, but it
should be mentioned.
> You could also try tightly timing the routine, but I'm not sure how
> precise you can get with the 1541 hardware. If you just went straight
> around the track 4 times transferring bits the whole way, you would
> get bitshifted data, etc. since the track length might not even be a
> multiple of 8 bits long. :)
That's exactly what Per has found a solution for. To get overlapped
alignment data for a full track, he's going to do seven reads, not
four.
> Anyway, if you can get a routine that transfers a track this way, I
> would be happy to help however I can to figure out how to process the
> data. If it works, it would open up the ability to image disks to a
> wider audience.
I believe transfer to the C64 should work, and also that data
reassembly should be fairly simple (on a PC - it would probably be
very slow on a C64, especially with bit-shifting).
So, the only open question to me is whether it's possible to transfer
the data directly to a PC. This is something I know exactly nothing
about, and I'm looking forward to seeing more knowledgeable people
work that one out :-)
--
Dag Lem <resid@nimrod.no>
|
|
|
|
| Re: New release 0.4.0 of opencbm (cbm4win/cbm4linux) [message #142335 is a reply to message #142331] |
Sun, 07 May 2006 14:39 |
|
|
Originally posted by: Dag Lem
MagerValp <MagerValp@cling.gu.se> writes:
>>>> >> "DL" == Dag Lem <resid@nimrod.no> writes:
>
> DL> Isn't it possible to keep the ATN A bit out of the equation by
> DL> toggling bit 4 in the data direction register $1803?
>
> Since we're talking custom IEC code here, can't the PC or C64 just
> leave ATN alone? :)
I think what Wolfgang's main worry was that if the 1541 sets the ATN A
bit, the DATA OUT bit would be clobbered by the circuitry connecting
ATN A to DATA OUT. I believe this can be solved by setting the data
direction register for ATN A to "input". I've seen the ASL trick used
in at least one disk turbo, so it should work.
But yes, the PC and the C64 should just leave ATN alone :-)
--
Dag Lem <resid@nimrod.no>
|
|
|
|
| Re: Serial line Burstnibbler, Was: [...] 0.4.0 of opencbm [...] [message #142336 is a reply to message #142334] |
Sun, 07 May 2006 14:51 |
Pete Rittwage
Messages: 444
Registered: March 2005
Karma: 0
|
Senior Member |
|
|
Dag Lem wrote:
> Pete Rittwage <rcade@rittwage.com> writes:
>
>
>> One problem I see right away in taking only 2 bits at a time, is how
>> do you know how to piece this back together on the other end into a
>> valid track cycle? Currently, we take 8192 bytes all in one sweep,
>> which allows for differing amounts of overlap in the GCR data at
>> different bitrates, and then I run it through a series of different
>> algorithms that attempt to detect this overlap and establish how the
>> track was "probably" appears on the disk.
>
>
> I think the crux of Per's idea is to do overlapped full track reads,
> like this:
>
> 1: transfer bits 0, 1
> 2: transfer bits 1, 2
> 3: transfer bits 2, 3
> 4: transfer bits 3, 4
> 5: transfer bits 4, 5
> 6: transfer bits 5, 6
> 7: transfer bits 6, 7
>
> He can then overlay and (bit)shift the data until it all lines up to
> the same 8192 bytes you'd read in one go using a parallel cable.
But you don't have that data to compare to. I guess you could read a
small chunk of data into drive memory, transfer it to the host, then
bitshift until you match the data. Might work, but won't cover
protection-only signature tracks.
>
> Note that in general it isn't enough to simply shift bytes, if there
> are no sync marks you're not guaranteed to get byte aligned reads. I
> don't know whether that's anything to worry about in practice, but it
> should be mentioned.
>
There are lots of protections that use syncless tracks, unfortunately.
>> You could also try tightly timing the routine, but I'm not sure how
>> precise you can get with the 1541 hardware. If you just went straight
>> around the track 4 times transferring bits the whole way, you would
>> get bitshifted data, etc. since the track length might not even be a
>> multiple of 8 bits long. :)
>
>
> That's exactly what Per has found a solution for. To get overlapped
> alignment data for a full track, he's going to do seven reads, not
> four.
>
I don't know if single bit of overlap is enough to help you. You don't
know what you're supposed to be looking at unless you compare it to
something. :)
I hope I am not coming across as knocking the idea, because I would love
to see it work. I think that transferring the bits like this in
combination with maybe a short sample of the track data to line up the
data could work for a lot of disks.
-
Pete Rittwage
C64 Preservation Project
http://c64preservation.com
|
|
|
|
| Re: Serial line Burstnibbler, Was: [...] 0.4.0 of opencbm [...] [message #142337 is a reply to message #142332] |
Sun, 07 May 2006 15:23 |
Wolfgang Moser
Messages: 632
Registered: July 2003
Karma: 0
|
Senior Member |
|
|
Hi Pete,
Pete Rittwage schrieb:
>
> One problem I see right away in taking only 2 bits at a time, is how do
> you know how to piece this back together on the other end into a valid
> track cycle? Currently, we take 8192 bytes all in one sweep, which
Per always wants to read one of the bits again
on the next cycle. If one byte from DC_DATA/1C01
consists of the bits:
AxBxCxDxExFxFxHx
Then on the first run he transmits
A1, B1 ... A2, B2 ... A3, B3 ...
On the next cycle:
B1, C1 ... B2, C2 ... B3, C3 ...
Thus he later can adjust both bitstreams
by trying to match the both B-streams.
> You could have some sort of "bookmark" pattern you scanned for on the
> track to line it up, but that will fail on many protection tracks that
> are not anything like standard DOS tracks. They might be all sync, all
The two worst things I currently can think of
that would make the bitstream matching somewhat
difficult were in-track bitrate changes and
weak bit areas.
Both protection techniques would stretch or
squeeze the data streams so that only shorter
runs of it can be matched perfectly. You would
need to do re-stretching to compensate. These
would be the higher educated things Per would
have to solve one day.
If you ask me, I would be very impressed, if
Per could come up with showing that his technique
does work on _unprotected_ disks first.
> multi-byte signature pattern that is repeated many times. In these, you
> cannot bookmark a certain byte pattern. The only way this would seem to
> work is on standard DOS sectors or something very close.
Hmmm, didn't I mention to do some sort of
histogramm with entries of bitstream pieces of
32 bits or so to find such unique patterns?
This may also be helpful with Per's technique
if you're into statistics, but who really is? ;-)
> bitshifted data, etc. since the track length might not even be a
> multiple of 8 bits long. :)
This and these protections loosing frame sync
all day long (latest VMAX ?) is a fact that
shows up some limits; to be solved later ;-)
Womo
|
|
|
|
| Re: Serial line Burstnibbler, Was: [...] 0.4.0 of opencbm [...] [message #142338 is a reply to message #142336] |
Sun, 07 May 2006 15:26 |
|
|
Originally posted by: Dag Lem
Pete Rittwage <rcade@rittwage.com> writes:
> I don't know if single bit of overlap is enough to help you. You
> don't know what you're supposed to be looking at unless you compare it
> to something. :)
>
> I hope I am not coming across as knocking the idea, because I would
> love to see it work. I think that transferring the bits like this in
> combination with maybe a short sample of the track data to line up the
> data could work for a lot of disks.
Remember that all seven full track reads are linked together with
overlapping bits.
Even though there may be several possibilites of joining, say, bits
0,1,2, this join will have to match a join of bits 1,2,3. And so on.
When it all lines up, the bits will be correctly aligned, as far as I
can tell.
Sure, you can construct cases where the sceme falls apart, but I
believe the possibility of that happening with a real disk image is
very close to zero.
One obvious case is to fill an entire track with all zeros or all ones
in just one bit. That would give us absolutely no clue for alignment.
After all, there is approximately 8KiB of data to line up, and I'd be
very surprised if you come up with a real life disk image where the
bit patterns are so repetitive that it's possible to misalign the
data.
Do you really see the possibility of this happening in practice? I
simply don't have the knowledge about all the strange disk images out
there to give a definitive answer.
--
Dag Lem <resid@nimrod.no>
|
|
|
|
| Re: Serial line Burstnibbler, Was: [...] 0.4.0 of opencbm [...] [message #142339 is a reply to message #142332] |
Sun, 07 May 2006 15:27 |
Wolfgang Moser
Messages: 632
Registered: July 2003
Karma: 0
|
Senior Member |
|
|
Hi Pete,
Pete Rittwage schrieb:
>
> One problem I see right away in taking only 2 bits at a time, is how do
> you know how to piece this back together on the other end into a valid
> track cycle? Currently, we take 8192 bytes all in one sweep, which
Per always wants to read one of the bits again
on the next cycle. If one byte from DC_DATA/1C01
consists of the bits:
AxBxCxDxExFxFxHx
Then on the first run he transmits
A1, B1 ... A2, B2 ... A3, B3 ...
On the next cycle:
B1, C1 ... B2, C2 ... B3, C3 ...
Thus he later can adjust both bitstreams
by trying to match the both B-streams.
> You could have some sort of "bookmark" pattern you scanned for on the
> track to line it up, but that will fail on many protection tracks that
> are not anything like standard DOS tracks. They might be all sync, all
The two worst things I currently can think of
that would make the bitstream matching somewhat
difficult were in-track bitrate changes and
weak bit areas.
Both protection techniques would stretch or
squeeze the data streams so that only shorter
runs of it can be matched perfectly. You would
need to do re-stretching to compensate. These
would be the higher educated things Per would
have to solve one day.
If you ask me, I would be very impressed, if
Per could come up with showing that his technique
does work on _unprotected_ disks first.
> multi-byte signature pattern that is repeated many times. In these, you
> cannot bookmark a certain byte pattern. The only way this would seem to
> work is on standard DOS sectors or something very close.
Hmmm, didn't I mention to do some sort of
histogramm with entries of bitstream pieces of
32 bits or so to find such unique patterns?
This may also be helpful with Per's technique
if you're into statistics, but who really is? ;-)
> bitshifted data, etc. since the track length might not even be a
> multiple of 8 bits long. :)
This and these protections loosing frame sync
all day long (latest VMAX ?) is a fact that
shows some limits
>
> Anyway, if you can get a routine that transfers a track this way, I
> would be happy to help however I can to figure out how to process the
> data. If it works, it would open up the ability to image disks to a
> wider audience.
>
> -
> Pete Rittwage
> C64 Preservation Project
> http://c64preservation.com
|
|
|
|
| Re: Serial line Burstnibbler, Was: [...] 0.4.0 of opencbm [...] [message #142340 is a reply to message #142338] |
Sun, 07 May 2006 15:33 |
Pete Rittwage
Messages: 444
Registered: March 2005
Karma: 0
|
Senior Member |
|
|
Dag Lem wrote:
> Pete Rittwage <rcade@rittwage.com> writes:
>
>
>> I don't know if single bit of overlap is enough to help you. You
>> don't know what you're supposed to be looking at unless you compare it
>> to something. :)
>>
>> I hope I am not coming across as knocking the idea, because I would
>> love to see it work. I think that transferring the bits like this in
>> combination with maybe a short sample of the track data to line up the
>> data could work for a lot of disks.
>
>
> Remember that all seven full track reads are linked together with
> overlapping bits.
>
> Even though there may be several possibilites of joining, say, bits
> 0,1,2, this join will have to match a join of bits 1,2,3. And so on.
>
> When it all lines up, the bits will be correctly aligned, as far as I
> can tell.
>
> Sure, you can construct cases where the sceme falls apart, but I
> believe the possibility of that happening with a real disk image is
> very close to zero.
>
> One obvious case is to fill an entire track with all zeros or all ones
> in just one bit. That would give us absolutely no clue for alignment.
>
> After all, there is approximately 8KiB of data to line up, and I'd be
> very surprised if you come up with a real life disk image where the
> bit patterns are so repetitive that it's possible to misalign the
> data.
>
> Do you really see the possibility of this happening in practice? I
> simply don't have the knowledge about all the strange disk images out
> there to give a definitive answer.
>
I think it could work with a lot of disks- you might do better than
Turbonibbler. Maybe 3% of disks would be tough to transfer this way
because of syncless tracks or signature tracks or weak bits, but we
always still have other techniques for this data we are already using.
-
Pete Rittwage
C64 Preservation Project
http://c64preservation.com
|
|
|
|
| Re: Serial line Burstnibbler, Was: [...] 0.4.0 of opencbm [...] [message #142341 is a reply to message #142337] |
Sun, 07 May 2006 15:55 |
|
|
Originally posted by: Dag Lem
Wolfgang Moser <wnhp@d81.de.invalid> writes:
> The two worst things I currently can think of
> that would make the bitstream matching somewhat
> difficult were in-track bitrate changes and
> weak bit areas.
> Both protection techniques would stretch or
> squeeze the data streams so that only shorter
> runs of it can be matched perfectly. You would
> need to do re-stretching to compensate. These
> would be the higher educated things Per would
> have to solve one day.
Ugh. I can see that my view of disk images is a bit simplistic. Just
out of interest, how do you handle bitrate changes today? Do you
manage to reassemble the track automatically by reading the track
several times, or is it necessary to extend a helping hand to the
nibbler?
I believe Per's idea could still work, even though admittedly it's
getting quite a bit more difficult to get things right.
> If you ask me, I would be very impressed, if
> Per could come up with showing that his technique
> does work on _unprotected_ disks first.
Go for it, Per! :-)
--
Dag Lem <resid@nimrod.no>
|
|
|
|
| Re: Serial line Burstnibbler, Was: [...] 0.4.0 of opencbm [...] [message #142342 is a reply to message #142340] |
Sun, 07 May 2006 16:06 |
|
|
Originally posted by: Dag Lem
Pete Rittwage <rcade@rittwage.com> writes:
> I think it could work with a lot of disks- you might do better than
> Turbonibbler. Maybe 3% of disks would be tough to transfer this way
> because of syncless tracks or signature tracks or weak bits, but we
> always still have other techniques for this data we are already using.
Syncless tracks should be OK if bit-shifting instead of byte-shifting
is used to align the data, right?
Weak bits are a problem, I can see that. However it should still be
doable, by allowing small shifts in the data and a few bit comparison
errors.
The bitrate changes Wolfgang is talking about seem harder to handle,
since you can only reliably match the parts written with the speed you
happen to be reading at. However I can think of harder problems.
Can you tell me a bit about signature tracks? To begin with, what is a
signature track? :-)
--
Dag Lem <resid@nimrod.no>
|
|
|
|
| Re: Serial line Burstnibbler, Was: [...] 0.4.0 of opencbm [...] [message #142343 is a reply to message #142342] |
Sun, 07 May 2006 16:36 |
Pete Rittwage
Messages: 444
Registered: March 2005
Karma: 0
|
Senior Member |
|
|
Dag Lem wrote:
> Pete Rittwage <rcade@rittwage.com> writes:
>
>
>> I think it could work with a lot of disks- you might do better than
>> Turbonibbler. Maybe 3% of disks would be tough to transfer this way
>> because of syncless tracks or signature tracks or weak bits, but we
>> always still have other techniques for this data we are already using.
>
>
> Syncless tracks should be OK if bit-shifting instead of byte-shifting
> is used to align the data, right?
>
> Weak bits are a problem, I can see that. However it should still be
> doable, by allowing small shifts in the data and a few bit comparison
> errors.
>
> The bitrate changes Wolfgang is talking about seem harder to handle,
> since you can only reliably match the parts written with the speed you
> happen to be reading at. However I can think of harder problems.
I don't have any disks I have run across that change bitrates mid-track.
It's a technique that's been discussed, but I've never seen it on a
disk to this point.
>
> Can you tell me a bit about signature tracks? To begin with, what is a
> signature track? :-)
>
It's just a track that's written in a funny way that is detected by copy
protection- they typically don't contain data. EA used "Pirateslayer"
which you can read about on my site- it's basically a syncless track
with a certain byte pattern that is searched for. Other protections
used similar techniques either searching for only one byte or a couple
in a certain order.
-
Pete Rittwage
C64 Preservation Project
http://c64preservation.com
|
|
|
|
| Re: Serial line Burstnibbler, Was: [...] 0.4.0 of opencbm [...] [message #142344 is a reply to message #142343] |
Sun, 07 May 2006 17:47 |
|
|
Originally posted by: Dag Lem
Pete Rittwage <rcade@rittwage.com> writes:
> Dag Lem wrote:
>
>> Can you tell me a bit about signature tracks? To begin with, what is
>> a
>> signature track? :-)
>>
>
> It's just a track that's written in a funny way that is detected by
> copy protection- they typically don't contain data. EA used
> "Pirateslayer" which you can read about on my site- it's basically a
> syncless track with a certain byte pattern that is searched for.
> Other protections used similar techniques either searching for only
> one byte or a couple in a certain order.
Ah, OK. If the entire track contains just one or very few consecutive
bytes, then I agree that finding the correct alignment may be
problematic. In this case, you may get several possible alignments,
and you can't really tell which one is correct.
I suppose some heuristics can be used here, e.g. set bits should most
probably be concentrated into as few bytes as possible. Granted, this
won't be foolproof, but perhaps it can drive down the number of
uncopyable disks to a handful or so. I haven't analyzed whether it
would work for "Pirateslayer", perhaps you want to look into that ;-)
--
Dag Lem <resid@nimrod.no>
|
|
|
|
| Re: Serial line Burstnibbler, Was: [...] 0.4.0 of opencbm [...] [message #142345 is a reply to message #142342] |
Sun, 07 May 2006 18:15 |
Wolfgang Moser
Messages: 632
Registered: July 2003
Karma: 0
|
Senior Member |
|
|
Hello Pete, Dag,
Dag Lem schrieb:
> Pete Rittwage <rcade@rittwage.com> writes:
>
>> I think it could work with a lot of disks- you might do better than
>> Turbonibbler. Maybe 3% of disks would be tough to transfer this way
In fact, Turbonibbler 4.0 should definetly be beaten by
this technique.
>> because of syncless tracks or signature tracks or weak bits, but we
>> always still have other techniques for this data we are already using.
>
> Syncless tracks should be OK if bit-shifting instead of byte-shifting
> is used to align the data, right?
Let's wait and see, the difficult part is that you
comepletely miss frame sync and therefore you never
know which bits of a byte you are actually masking
out (reading) and transferring to the host.
Here you would at least have to do some data
analysations with_in_ the drive -- maybe my beloved
histogramm functions -- to find a unique pattern
that then hopefully could be used for manually
framing.
Manually framing, Pete said, is done with the
California Games protection, right? They're changing
the bitrate register value, so that the deserializer
gets out of sync very often. Once you see an
expected bit/byte pattern you program the correct
bitrate settign and got into sync with the expected
byte frames.
> The bitrate changes Wolfgang is talking about seem harder to handle,
> since you can only reliably match the parts written with the speed you
> happen to be reading at. However I can think of harder problems.
Well, as Pete explains, they seem to not be in use
although it seems doubtful to me, if Pete would be
able to detect such bitrate changed areas by only
analysing data that was MNibbed. Data recorded with
the wrong bitrate _could_ look like a weak bit area.
As long as there're only five titles around using
such a scheme, nothing is better than remastering it
by hand (G64 is capable in doing so).
Womo
|
|
|
|
| Re: Serial line Burstnibbler, Was: [...] 0.4.0 of opencbm [...] [message #142346 is a reply to message #142345] |
Mon, 08 May 2006 03:12 |
|
|
Originally posted by: Dag Lem
Wolfgang Moser <wnhp@d81.de.invalid> writes:
> Hello Pete, Dag,
>
> Dag Lem schrieb:
>> Pete Rittwage <rcade@rittwage.com> writes:
>>> because of syncless tracks or signature tracks or weak bits, but we
>>> always still have other techniques for this data we are already using.
>> Syncless tracks should be OK if bit-shifting instead of byte-shifting
>> is used to align the data, right?
>
> Let's wait and see, the difficult part is that you
> comepletely miss frame sync and therefore you never
> know which bits of a byte you are actually masking
> out (reading) and transferring to the host.
True, but if you bit-shift instead of byte-shift the data for
alignment, you should end up with the same result as by using a
parallel cable.
I seem to remember someone (you? Pete?) mentioning that VICE does byte
based disk reads, and that data in some cases has to be bit-shifted in
order to make the images work on VICE. Is this what you're thinking
about? In this case we're not really talking about a new problem.
> Manually framing, Pete said, is done with the
> California Games protection, right? They're changing
> the bitrate register value, so that the deserializer
> gets out of sync very often. Once you see an
> expected bit/byte pattern you program the correct
> bitrate settign and got into sync with the expected
> byte frames.
Uh oh :-/
--
Dag Lem <resid@nimrod.no>
|
|
|
|
| Re: New release 0.4.0 of opencbm (cbm4win/cbm4linux) [message #142347 is a reply to message #142322] |
Mon, 08 May 2006 04:17 |
|
|
Originally posted by: Dag Lem
Dag Lem <resid@nimrod.no> writes:
> Perhaps this will do the trick?
>
> ...
> 2 bvc *
> 4 clv
> 6 eor #$0a
> 10 sta $1800
> 14 ldx $1c01
> 18 lda bittab,x
> 22 sta $1800
> ...
Coming to think of it, perhaps you can even manage to push three bits
across, like this:
...
2 and #$02 ; isolate bit1 from previous transfer
4 bvc *
8 ldx $1c01
12 eor bittab,x ; bittab[x].bit1 = 1 => previous bit1 is inverted
16 sta $1800 ; transfer "sync" bit + one data bit
18 clv
20 asl
24 sta $1800 ; transfer two data bits
...
At 24 cycles, not considering (unrolled) loop overhead, you would be
pushing it, I guess. And, if I understand Wolfgang correctly, relying
on exact timing is probably only OK when doing transfers to a C64. It
would mean that you could get away with four track reads instead of
seven, though :-)
Wolfgang: Do you know how many cycles the 1541 can "skip" during a
byte read? I.e. what is the absolute minimum of cycles spent on
reading one byte?
--
Dag Lem <resid@nimrod.no>
|
|
|
|
| Re: Serial line Burstnibbler, Was: [...] 0.4.0 of opencbm [...] [message #142348 is a reply to message #142346] |
Mon, 08 May 2006 08:10 |
Pete Rittwage
Messages: 444
Registered: March 2005
Karma: 0
|
Senior Member |
|
|
Dag Lem wrote:
> Wolfgang Moser <wnhp@d81.de.invalid> writes:
>
>
>> Hello Pete, Dag,
>>
>> Dag Lem schrieb:
>>
>>> Pete Rittwage <rcade@rittwage.com> writes:
>>>
>>>> because of syncless tracks or signature tracks or weak bits, but we
>>>> always still have other techniques for this data we are already using.
>>>
>>> Syncless tracks should be OK if bit-shifting instead of byte-shifting
>>> is used to align the data, right?
>>
>> Let's wait and see, the difficult part is that you
>> comepletely miss frame sync and therefore you never
>> know which bits of a byte you are actually masking
>> out (reading) and transferring to the host.
>
>
> True, but if you bit-shift instead of byte-shift the data for
> alignment, you should end up with the same result as by using a
> parallel cable.
>
> I seem to remember someone (you? Pete?) mentioning that VICE does byte
> based disk reads, and that data in some cases has to be bit-shifted in
> order to make the images work on VICE. Is this what you're thinking
> about? In this case we're not really talking about a new problem.
>
>
It's a similar problem. Without framing we have a 1/8 chance of getting
it right with a byte-read track and then we must know what the
protection is looking for to shift it, so it's not a new problem. It's
knowing how the bits would fit together in relation to each other that
would be a new problem. You're not talking about just shifting, but
unscrambling. :)
>> Manually framing, Pete said, is done with the
>> California Games protection, right? They're changing
>> the bitrate register value, so that the deserializer
>> gets out of sync very often. Once you see an
>> expected bit/byte pattern you program the correct
>> bitrate settign and got into sync with the expected
>> byte frames.
>
>
> Uh oh :-/
>
Not that I've ever seen, don't worry. :)
-
Pete Rittwage
http://rittwage.com
|
|
|
|
|
|
| Re: Serial line Burstnibbler, Was: [...] 0.4.0 of opencbm [...] [message #142352 is a reply to message #142336] |
Mon, 08 May 2006 11:38 |
MagerValp
Messages: 847
Registered: April 2012
Karma: 0
|
Senior Member |
|
|
>>>> > "PR" == Pete Rittwage <rcade@rittwage.com> writes:
PR> But you don't have that data to compare to. I guess you could
PR> read a small chunk of data into drive memory, transfer it to the
PR> host, then bitshift until you match the data.
Excellent idea! You could even do a hybrid, where you read 8 kB of raw
data in 1 kB chunks, like a normal nibbler would, then read maybe one
or two 2-bit tracks using my method, and then just use the 2-bit data
to line up the byte chunks.
PR> I hope I am not coming across as knocking the idea, because I
PR> would love to see it work.
On the contrary, your comments are very useful. It's always good to
have flaws pointed out before you begin, so you can work your way
around them.
--
___ . . . . . + . . o
_|___|_ + . + . + . Per Olofsson, arkadspelare
o-o . . . o + MagerValp@cling.gu.se
- + + . http://www.cling.gu.se/~cl3polof/
|
|
|
|
|
|
| Re: New release 0.4.0 of opencbm (cbm4win/cbm4linux) [message #142354 is a reply to message #142347] |
Mon, 08 May 2006 12:17 |
Wolfgang Moser
Messages: 632
Registered: July 2003
Karma: 0
|
Senior Member |
|
|
Hello Dag, you asked:
> ...
> 2 and #$02 ; isolate bit1 from previous transfer
> 4 bvc *
> 8 ldx $1c01
> 12 eor bittab,x ; bittab[x].bit1 = 1 => previous bit1 is inverted
> 16 sta $1800 ; transfer "sync" bit + one data bit
> 18 clv
> 20 asl
> 24 sta $1800 ; transfer two data bits
> ...
>
> At 24 cycles, not considering (unrolled) loop overhead, you would be
> pushing it, I guess. And, if I understand Wolfgang correctly, relying
> on exact timing is probably only OK when doing transfers to a C64. It
> would mean that you could get away with four track reads instead of
> seven, though :-)
>
> Wolfgang: Do you know how many cycles the 1541 can "skip" during a
> byte read? I.e. what is the absolute minimum of cycles spent on
> reading one byte?
I don't know if I understand you right. Is it about
the distance in time, where the Byte Ready signal
occurs (V flag set)?
Maybe an excerpt from the comments I wrote to one
subroutine of OpenCBM's cbmforng does answer your
question? It repeats some things I wrote on an
earlier post here.
OpenCBM-0.4.0/cbmforng/cbmfsubs.i65:
> [...]
> ; If the previous write to DC_DATA before calling this funtion was
> ; nearly perfectly in sync to the Byte-Ready-Signal, then this
> ; single write to DC_DATA _may_ be delayed by up to 40 cycles
> ; despite the theoretical minimum value of less than 26 cycles.
> ; But the delayed write (15 cycles hanging over) needs to be
> ; compensated by at least two additional runs of the loop in
> ; "BufferedWriteLoop".
> ; Such a single write delay is possible because the written value
> ; is latched into the serial shift register with the _next_
> ; Byte-Ready-Signal, which means a byte needs to be written to
> ; DC_DATA just _before_ the next Byte-Ready-Signal happens.
> ;
> ;
> ; clock cycle (cylic)
> ; 11111111111222222 11111111111222222
> ; 123456789012345678901234561234567890123456789012345612345678 9
> ;
> ;clock pulse of the Byte-Ready signal, byte at PA is latched
> ; 000100000000000000000000000001000000000000000000000000010000 0000
> ; | | |
> ; | | |
> ;possible exits in time of the bvc * loop
> ; BBB11 11
> ; BBB11 11
> ; BBB11 . 11 .
> ; | . | . |
> ; | . | . |
> ; V-Flag resetting (done via SBC) .
> ; CC CC
> ; | . . | . . |
> ; normal store to DC_DATA (in time) delayed store (last possible)
> ; . SSSS . SSSS
> ; | . . . | . . |
> ; 1234. . . 1111111111222222222233333333334. .
> ; 123456789012345678901234567890123456789012345678
> ;
> ; The abstracted schema above was derived from real disk drive tests.
> ; The maximum delay of 40 cycles can only be reached, if the V flag
> ; reset is made before the store to DC_DATA. If the store to DC_DATA
> ; comes first, then the maximum delay for the store would only be 39
> ; cycles.
> [...]
In the diagram above the meaning of some abbreviations is:
B -- BVC * actually is executing a branch
l -- BVC * actually is executing a non-branch
C -- Execution cycles of CLV
S -- Execution cycles of sta DC_DATA
Furthermore I hope that this diagram better
explains to you, why I would like to see the clv
beeing executed earlier ;-)
Womo
|
|
|
|
| Re: Serial line Burstnibbler, Was: [...] 0.4.0 of opencbm [...] [message #142355 is a reply to message #142352] |
Mon, 08 May 2006 12:48 |
Wolfgang Moser
Messages: 632
Registered: July 2003
Karma: 0
|
Senior Member |
|
|
Hello Per, Pete, Dag,
MagerValp wrote:
> "PR" == Pete Rittwage <rcade@rittwage.com> writes:
> PR> But you don't have that data to compare to. I guess you could
> PR> read a small chunk of data into drive memory, transfer it to the
> PR> host, then bitshift until you match the data.
>
> Excellent idea! You could even do a hybrid, where you read 8 kB of raw
> data in 1 kB chunks, like a normal nibbler would, then read maybe one
> or two 2-bit tracks using my method, and then just use the 2-bit data
> to line up the byte chunks.
"Hybrid" smells really goood. If you then add timer-
driven techniques for selecting the chunks in time
(assuming that disk rotation is not changing too much),
you even would not need _any_ marker bytes from the
track (like SYNCs). You then in fact can align the
data by two independent correction vectors (assuming
orthogonality):
* overlapping of all the chunks by maybe
reading 10K in sum, trying to find the
correct alignment by bitshifting
* aligning all the chunks with the help of
your serial burst algorithm
When the first technique fails due to weak bits just
in the overlap area, the second my help.
If the second method fails due to too regular bit
patterns (long SYNC), well..., hmmm.... Then the first
doesn't help also, <sigh> Well, then you would need
have to burst several more bit"lines" perhaps.
Maybe other data synchronization methods can be used
in that hybrid, too.
> PR> I hope I am not coming across as knocking the idea, because I
> PR> would love to see it work.
>
> On the contrary, your comments are very useful. It's always good to
> have flaws pointed out before you begin, so you can work your way
> around them.
But sometimes it stops you from trying out and maybe
experiencing that it could have been done. As I said
earlier, if you can proof that the technique works
with unprotected disks, you (we ?) may sucessively
extend it to read further and further protections.
Just the same as Pete is doing: Looking at difficult
protections by hand, improving the postprocessing and
data sampling techniques and thus automating the
process.
Btw. Pete, there's one thing that maybe could be used
sometime to detect for in-track-speedchanges:
Let a timer run in the floppy. With every Byte-Ready
signal don't read the DC_DATA register, but the low
value of the timer (counting clock cycles) and
transmit it with the parallel burst protocol. On the
PC side you then can tell for every 8 Bits window,
at which speed it was written.
With some subsampling techniques (or statistics
alternatively) you would be able to go below the 8
bits window size and may determine the exact position
where the bitrate change occured.
Some digital processing techniques would allow to
find positions in the stream, where the Byte Ready
signal got in perfect sync to the BVC * loop, for
example if you got count distances of:
28, 28, 28, 28, 28, 28, 28, 27, 28, 28, 28, ...
Exactly at the position where only 27 cycles were
counted you got in perfect sync. As another example:
30, 30, 30, 30, 30, 31, 30, 30, 30, 30, 30, ...
Just before the 31 cycles are needed, on the last
30, you got in perfect sync to the Byte Ready signal.
These perfect syncs may be used as definitve
reference points for determining absolute bitrate
speeds. In combination with the known or measured out
RPM value of your current drive you would be able to
determine at which exact bitrate the disk was
_recorded_!
But I better stop producing more and more vaporware,
I'll never got implemented it. Or, if I got
implemented something (cbmforng), I don't get it
documented. I want to work on the latter one first,
so that people get able to learn from it and start
doing things on their own.
Womo
|
|
|
|
| Re: New release 0.4.0 of opencbm (cbm4win/cbm4linux) [message #142356 is a reply to message #142354] |
Mon, 08 May 2006 16:01 |
|
|
Originally posted by: Dag Lem
Wolfgang Moser <wnhp@d81.de.invalid> writes:
> Hello Dag, you asked:
>> Wolfgang: Do you know how many cycles the 1541 can "skip" during a
>> byte read? I.e. what is the absolute minimum of cycles spent on
>> reading one byte?
>
> I don't know if I understand you right. Is it about
> the distance in time, where the Byte Ready signal
> occurs (V flag set)?
What I'm really interested in is what kind of average safety margin is
needed on the 26 cycles in order to ensure that you can keep up with
incoming bytes. Yes, I suppose this is the same as the distance in
time between each time the V flag is set.
You've explained all about writing, but as far as I understand you
won't get below 26 cycles there - please correct me if I'm wrong. On
reading, however, rumour has it that you may get incoming bytes at a
faster rate, since set bits in the GCR scheme are used for
synchronization.
What I was wondering is really:
a) Is there a theroretical limit to how fast one byte can be delivered?
b) What is the practical maximum limit to how many cycles we can spend
(on average) in code reading one byte? Approximately 25 cycles
(3.8% speedup)?
> Furthermore I hope that this diagram better
> explains to you, why I would like to see the clv
> beeing executed earlier ;-)
Hmm, your diagram is a bit too much for me this evening, sorry :-)
Are you saying that it's not sufficient to ensure that you clear the V
flag within a time frame of approximately 25 cycles after it was set?
I don't quite understand how this could be a problem - I assumed that
clearing the V flag didn't affect the circuitry other than making it
possible to discern the next time the V flag is set. Perhaps that's a
wrong assumption?
--
Dag Lem <resid@nimrod.no>
|
|
|
|
| Re: Serial line Burstnibbler, Was: [...] 0.4.0 of opencbm [...] [message #142357 is a reply to message #142355] |
Mon, 08 May 2006 16:42 |
Pete Rittwage
Messages: 444
Registered: March 2005
Karma: 0
|
Senior Member |
|
|
Wolfgang Moser wrote:
> Hello Per, Pete, Dag,
>
> MagerValp wrote:
>
>> "PR" == Pete Rittwage <rcade@rittwage.com> writes:
>> PR> But you don't have that data to compare to. I guess you could
>> PR> read a small chunk of data into drive memory, transfer it to the
>> PR> host, then bitshift until you match the data.
>>
>> Excellent idea! You could even do a hybrid, where you read 8 kB of raw
>> data in 1 kB chunks, like a normal nibbler would, then read maybe one
>> or two 2-bit tracks using my method, and then just use the 2-bit data
>> to line up the byte chunks.
>
Did any serial nibblers actually handle it that way? I haven't
disassembled them widely (Joe Forster might know this better than
anyone). It seems they just read in a sector at a time ignoring the
headers, then reconstructed it. All the serial nibblers seemed to miss
anything in the gaps (head or tail) which leads me to believe they only
reproduced the actual valid sector data.
>
> "Hybrid" smells really goood. If you then add timer-
> driven techniques for selecting the chunks in time
> (assuming that disk rotation is not changing too much),
> you even would not need _any_ marker bytes from the
> track (like SYNCs). You then in fact can align the
> data by two independent correction vectors (assuming
> orthogonality):
> * overlapping of all the chunks by maybe
> reading 10K in sum, trying to find the
> correct alignment by bitshifting
> * aligning all the chunks with the help of
> your serial burst algorithm
>
> When the first technique fails due to weak bits just
> in the overlap area, the second my help.
> If the second method fails due to too regular bit
> patterns (long SYNC), well..., hmmm.... Then the first
> doesn't help also, <sigh> Well, then you would need
> have to burst several more bit"lines" perhaps.
>
> Maybe other data synchronization methods can be used
> in that hybrid, too.
>
>
>> PR> I hope I am not coming across as knocking the idea, because I
>> PR> would love to see it work.
>>
>> On the contrary, your comments are very useful. It's always good to
>> have flaws pointed out before you begin, so you can work your way
>> around them.
>
>
> But sometimes it stops you from trying out and maybe
> experiencing that it could have been done. As I said
> earlier, if you can proof that the technique works
> with unprotected disks, you (we ?) may sucessively
> extend it to read further and further protections.
> Just the same as Pete is doing: Looking at difficult
> protections by hand, improving the postprocessing and
> data sampling techniques and thus automating the
> process.
>
I think it's a great idea and will work. You should have thought of it
20 years ago. (or did you?) :)
> Btw. Pete, there's one thing that maybe could be used
> sometime to detect for in-track-speedchanges:
>
> Let a timer run in the floppy. With every Byte-Ready
> signal don't read the DC_DATA register, but the low
> value of the timer (counting clock cycles) and
> transmit it with the parallel burst protocol. On the
> PC side you then can tell for every 8 Bits window,
> at which speed it was written.
> With some subsampling techniques (or statistics
> alternatively) you would be able to go below the 8
> bits window size and may determine the exact position
> where the bitrate change occured.
> Some digital processing techniques would allow to
> find positions in the stream, where the Byte Ready
> signal got in perfect sync to the BVC * loop, for
> example if you got count distances of:
>
> 28, 28, 28, 28, 28, 28, 28, 27, 28, 28, 28, ...
>
> Exactly at the position where only 27 cycles were
> counted you got in perfect sync. As another example:
>
> 30, 30, 30, 30, 30, 31, 30, 30, 30, 30, 30, ...
>
> Just before the 31 cycles are needed, on the last
> 30, you got in perfect sync to the Byte Ready signal.
>
>
> These perfect syncs may be used as definitve
> reference points for determining absolute bitrate
> speeds. In combination with the known or measured out
> RPM value of your current drive you would be able to
> determine at which exact bitrate the disk was
> _recorded_!
>
Another great idea. I haven't seen a disk yet that changes speed zones
mid-track, but if we run across one, it will have to be programmed.
Actually it might be useful to recode the speed detection as it stands
now, actually. As it is, it samples the GCR at each of the bitrates and
uses the one with the least amount of invalid GCR. Not scientific at
all and certainly fails on tracks with long sync marks or real weak
areas from protection.
I'm not convinced that the newer Vorpal used this method. Reading at
the "normal" bitrates results in 100% valid GCR, but reading at the
"wrong" bitrate gives much bad GCR. They are just using bit patterns
that throw off the framing from sector to sector.
>
> But I better stop producing more and more vaporware,
> I'll never got implemented it. Or, if I got
> implemented something (cbmforng), I don't get it
> documented. I want to work on the latter one first,
> so that people get able to learn from it and start
> doing things on their own.
>
:)
-
Pete Rittwage
C64 Preservation Project
http://c64preservation.com
|
|
|
|
| Re: New release 0.4.0 of opencbm (cbm4win/cbm4linux) [message #142358 is a reply to message #142356] |
Mon, 08 May 2006 17:47 |
Wolfgang Moser
Messages: 632
Registered: July 2003
Karma: 0
|
Senior Member |
|
|
Hello Dag,
Dag Lem schrieb:
> Wolfgang Moser <wnhp@d81.de.invalid> writes:
>> I don't know if I understand you right. Is it about
>
> What I'm really interested in is what kind of average safety margin is
> needed on the 26 cycles in order to ensure that you can keep up with
> incoming bytes. Yes, I suppose this is the same as the distance in
> time between each time the V flag is set.
>
> You've explained all about writing, but as far as I understand you
> won't get below 26 cycles there - please correct me if I'm wrong. On
> reading, however, rumour has it that you may get incoming bytes at a
> faster rate, since set bits in the GCR scheme are used for
> synchronization.
Oh, now, go it. Yes, with "my" formatter you are
never below the 26 cycles because on writing you
are fixedly coupled to the system clock by the
programmable divider, 13 at the fastest bitrate
(16 for the slowest bitrate):
[4 is a fixed divider, 8 is for bits per byte]
16 MHz / 13 / 4 / 8 = 38462 Hz
=^= 26µs
And yes, at reading the bitclock and thus the
Byte Ready signal is derived from the 1-bits
magnetized on the disk.
If I argue things on it, I assume a "bad case"
clock rate which lets 25 cycles between two
reads.
> What I was wondering is really:
> a) Is there a theroretical limit to how fast one byte can be delivered?
> b) What is the practical maximum limit to how many cycles we can spend
> (on average) in code reading one byte? Approximately 25 cycles
> (3.8% speedup)?
Ok, well, you're asking for a worst-case
szenario? I never did it before, so let's do
some work on it. What would be a practical
worst-case szenario for the task of protection
busting?
Yepp, Harald Seeley's V-MAX!
Pete recently told me that drives need to be
adjusted to 297.5 rpm to become able to
remaster most V-MAX! titles.
Newer Vorpal protections need the drives
adjusted to 297 rpm for their long tracks.
297 rpm is also what I believe is the lower
tolerancy boundary for this drive type. I
assume a tolerancy of +/-1%, although there
were times where I assumed +/-3%.
Until now I never saw a written word from
Commodore on this, someone else may help out
here.
Well with a disk recorded at 297rpm with a
bitrate of 4/13 MHz and then played back at
303rpm, to which bitrate would this result
then and what is the number of cycles passed
away between two consecutive Byte Ready
signals?
Bitrate = ( 4 / 13 ) MHz * ( 303 / 297 )
= 0.314 MHz = 314 kHz
Cycles = 26 * ( 297 / 303 ) = 25.485
This is not too critical, you only need to
save one cycle. Aaaargh, one _more_ cycle ;-)
Ok, now let's assume a tolerancy of +/-3%:
Cycles = 26 * ( 290 / 310 ) = 24.322
Well, tough, but I would believe that the PLL
(not really a PLL) would not lock in at such
a high discrepancy since some bitrates may be
misinterpreted as other ones on the higher
speed zones.
Since there're always 2 cycles "distance"
between two neighbored speed zones (26, 28, 30
and 32 cycles; derivable from the dividers 13,
14, 15 and 16), I would never go below one
third of such a 2 cycle window. Calculating
back from an assumed lower cycles boundary of
25.333 cycles, what would the resulting
tolerancy be then?
26 * (( 300 - x )/( 300 + x )) = 25.333
26 * ( 300 - x ) = 25.333 * ( 300 + x )
26 * 300 - 26 * x = 25.333 * 300 + 25.333 * x
26 * 300 - 25.333 * 300 = 25.333 * x + 26 * x
( 26 - 25.333 ) * 300 = ( 25.333 + 26 ) * x
0.666 * 300 = 51.333 * x
x = 300 * 0.666 / 51.333 = 3.892 ~= 3.9
+/-3.9rpm means a procentual tolerancy of 1.3%.
Any more questions? ;-))
>> Furthermore I hope that this diagram better
>> explains to you, why I would like to see the clv
>> beeing executed earlier ;-)
>
> Hmm, your diagram is a bit too much for me this evening, sorry :-)
>
> Are you saying that it's not sufficient to ensure that you clear the V
> flag within a time frame of approximately 25 cycles after it was set?
> I don't quite understand how this could be a problem - I assumed that
> clearing the V flag didn't affect the circuitry other than making it
> possible to discern the next time the V flag is set. Perhaps that's a
> wrong assumption?
You should ensure in any way that the V flag is
cleared, _before_ the next Byte Ready signal is
generated. Otherwise you would clear the flag for
the just newly arrived drive. Since the following
BVC * then waits again, you will have lost one
byte.
My diagram is derived from extensively testing.
Weeks before the final version I was at 42 cycles
maximum delay, but depending on bvc-jitter and
the order of "clv/sta DC_DATA" you will get down
to 40 or even less only 39 cycles under certain
circumstances.
In the real implementation of the formatter I
never went higher than 36 cycles of delay,
letting a safety margin of 4 cycles. Since I
wanted to build a _reliable_ fast formatter in
the first place I found it no good idea to
optimize it to death (and because I couldn't
find a bunch of testers willing to format and
test some hundred disks each).
Womo
|
|
|
|
| Re: Serial line Burstnibbler, Was: [...] 0.4.0 of opencbm [...] [message #142359 is a reply to message #142357] |
Mon, 08 May 2006 17:55 |
Wolfgang Moser
Messages: 632
Registered: July 2003
Karma: 0
|
Senior Member |
|
|
Pete Rittwage schrieb:
> Wolfgang Moser wrote:
>> Btw. Pete, there's one thing that maybe could be used
>> sometime to detect for in-track-speedchanges:
>>
>> Let a timer run in the floppy. With every Byte-Ready
>> signal don't read the DC_DATA register, but the low
>> value of the timer (counting clock cycles) and
>> transmit it with the parallel burst protocol. On the
>> PC side you then can tell for every 8 Bits window,
>> at which speed it was written.
>> [... more much too boring blabla ;-) ...]
>
> Another great idea. I haven't seen a disk yet that changes speed zones
> mid-track, but if we run across one, it will have to be programmed.
You got me wrong. I wanted to propose an exact
technique to _detect_ such bitrate changes. I
believe that you did not see such disks,
because the current tools are not able to
detect them properly and don't show this to
you. Instead you see an area of some spare
weak bits, don't you think so?
> Actually it might be useful to recode the speed detection as it stands
> now, actually. As it is, it samples the GCR at each of the bitrates and
> uses the one with the least amount of invalid GCR. Not scientific at
> all and certainly fails on tracks with long sync marks or real weak
> areas from protection.
:-))
Womo
|
|
|
|
| Re: Serial line Burstnibbler, Was: [...] 0.4.0 of opencbm [...] [message #142360 is a reply to message #142359] |
Mon, 08 May 2006 20:12 |
Pete Rittwage
Messages: 444
Registered: March 2005
Karma: 0
|
Senior Member |
|
|
Wolfgang Moser wrote:
> Pete Rittwage schrieb:
>
>> Wolfgang Moser wrote:
>>
>>> Btw. Pete, there's one thing that maybe could be used
>>> sometime to detect for in-track-speedchanges:
>>>
>>> Let a timer run in the floppy. With every Byte-Ready
>>> signal don't read the DC_DATA register, but the low
>>> value of the timer (counting clock cycles) and
>>> transmit it with the parallel burst protocol. On the
>>> PC side you then can tell for every 8 Bits window,
>>> at which speed it was written.
>>> [... more much too boring blabla ;-) ...]
>>
>>
>> Another great idea. I haven't seen a disk yet that changes speed
>> zones mid-track, but if we run across one, it will have to be programmed.
>
>
> You got me wrong. I wanted to propose an exact
> technique to _detect_ such bitrate changes. I
> believe that you did not see such disks,
> because the current tools are not able to
> detect them properly and don't show this to
> you. Instead you see an area of some spare
> weak bits, don't you think so?
>
Such a thing could exist, so it will get coded sometime. :)
-
Pete Rittwage
C64 Preservation Project
http://c64preservation.com
|
|
|
|
| Re: Serial line Burstnibbler, Was: [...] 0.4.0 of opencbm [...] [message #142361 is a reply to message #142357] |
Tue, 09 May 2006 05:26 |
sta
Messages: 243
Registered: July 2003
Karma: 0
|
Senior Member |
|
|
Hi guys,
>>> Excellent idea! You could even do a hybrid, where you read 8 kB of raw
>>> data in 1 kB chunks, like a normal nibbler would, then read maybe one
>>> or two 2-bit tracks using my method, and then just use the 2-bit data
>>> to line up the byte chunks.
>
> Did any serial nibblers actually handle it that way? I haven't
> disassembled them widely (Joe Forster might know this better than
> anyone).
No, I haven't disassembled nibblers, only plain disk turbos... By the way,
a crazy idea came to my mind. Imagine this scenario:
You launch a special nibbler-style software and insert the disk to be
nibbled. The software seeks to track #1, keeps the motor running, does not
move the head and starts waiting. You remove the original disk, insert a
blank disk and tell the software to continue. The software formats the
track and thus calculates its length, within a precision of a few bytes
(not absolutely precisely, because of the small variations in the motor
speed all the time, right?). Because the head hasn't moved and the motor
hasn't stopped, you can safely assume that the same track on the original
disk has "exactly" (within the same precision) the same size. If the
software is not sure, it can format the track like five times and keep
the average track length.
You can also format the whole blank disk and calculate track sizes. Perhaps,
only for the first track of each speed zone (as we discussed in connection
with cbmforng).
Then re-insert the original disk and start really nibbling. From the speed
zone and the (very precise) track length in bytes, you can calculate track
lengths even if (parts of) the track is actually read in a different speed
(provided that you can detect different speeds).
Makes any sense?
Joe
--
KOVÁCS Balázs alias Joe Forster/STA sta@c64.org; http://sta.c64.org
Don't E-mail spam, HTML or uncompressed files! More contacts on homepage
|
|
|
|
| Re: Serial line Burstnibbler, Was: [...] 0.4.0 of opencbm [...] [message #142362 is a reply to message #142361] |
Tue, 09 May 2006 06:00 |
Wolfgang Moser
Messages: 632
Registered: July 2003
Karma: 0
|
Senior Member |
|
|
Hello Joe,
Joe Forster/STA schrieb:
> blank disk and tell the software to continue. The software formats the
> track and thus calculates its length, within a precision of a few bytes
> (not absolutely precisely, because of the small variations in the motor
> speed all the time, right?). Because the head hasn't moved and the motor
> [...]
to me this sounds like rpm calibration. I think this
could be done with the original disk that we want to
nibble as long as it contains a standard track 18.
There you are able to trigger on the SYNC and header
for block 18;0, which should be unique for the whole
track.
With the help of timers (here we better would create
a coupled pair of two 16-bit timers, but I don't know
if this does work with the one from the VIA chips as
with the CIA chips), a true passive measurement
routine can be created.
I wanted to write such a true passive high precision
rpm measurement routine for a long time; at least
since I discussed with Andreas Boose and others, if
the different track radiuses would result in
different frictional resistances...
Beside a simplified version in cbmforng (not timer
based) this all is vaporware as usual (for me ;-).
> Then re-insert the original disk and start really nibbling. From the speed
> zone and the (very precise) track length in bytes, you can calculate track
> lengths even if (parts of) the track is actually read in a different speed
> (provided that you can detect different speeds).
>
> Makes any sense?
I don't know yet if it would make some sense. Sure,
it will in fact help to make some well estimated
assumptions, where a track overlap should be started
and ended at.
On the other side, constantly monitoring a running
timer while you do the nibbling and saving timer
values in an internal table piece for piece (chunks
of GCR data) may help to better reconstruct the
consecutive stream of GCR data out of the chunks and
it may help to identify bitrate changes (in-track
ones), if these do exist in reality.
Womo
|
|
|
|
| Re: Serial line Burstnibbler, Was: [...] 0.4.0 of opencbm [...] [message #142363 is a reply to message #142361] |
Tue, 09 May 2006 17:33 |
Pete Rittwage
Messages: 444
Registered: March 2005
Karma: 0
|
Senior Member |
|
|
Joe Forster/STA wrote:
> Hi guys,
>
>
>>>> Excellent idea! You could even do a hybrid, where you read 8 kB of raw
>>>> data in 1 kB chunks, like a normal nibbler would, then read maybe one
>>>> or two 2-bit tracks using my method, and then just use the 2-bit data
>>>> to line up the byte chunks.
>>
>> Did any serial nibblers actually handle it that way? I haven't
>> disassembled them widely (Joe Forster might know this better than
>> anyone).
>
>
> No, I haven't disassembled nibblers, only plain disk turbos... By the way,
> a crazy idea came to my mind. Imagine this scenario:
>
> You launch a special nibbler-style software and insert the disk to be
> nibbled. The software seeks to track #1, keeps the motor running, does not
> move the head and starts waiting. You remove the original disk, insert a
> blank disk and tell the software to continue. The software formats the
> track and thus calculates its length, within a precision of a few bytes
> (not absolutely precisely, because of the small variations in the motor
> speed all the time, right?). Because the head hasn't moved and the motor
> hasn't stopped, you can safely assume that the same track on the original
> disk has "exactly" (within the same precision) the same size. If the
> software is not sure, it can format the track like five times and keep
> the average track length.
The data on the original is not necessarily the same size as what we can
write at any given bitrate. See below.
>
> You can also format the whole blank disk and calculate track sizes. Perhaps,
> only for the first track of each speed zone (as we discussed in connection
> with cbmforng).
>
> Then re-insert the original disk and start really nibbling. From the speed
> zone and the (very precise) track length in bytes, you can calculate track
> lengths even if (parts of) the track is actually read in a different speed
> (provided that you can detect different speeds).
>
> Makes any sense?
>
Hi Joe,
Yes, we do some of this already. When reading the disk, we calculate
the track lengths by analyzing more than one revolution of data and
searching for the overlap.
When writing it back, we calculate the length of each track by testing
on the first track of each speed zone, just like you mentioned. :)
We then adjust the data to fit, where possible. We shorten syncs,
shorten gaps, and remove weak bit areas to get it to fit if it's a long
track. We also pad short tracks so that no data is left behind from a
non-blank disk.
When reading a track, the drive circuitry has a certain tolerance where
it can read data that was not written at the exact bitrate of a 1541
(like from a mastering device or a slow/fast drive). Sometimes the
drive just needs to be slowed down to get the data back on, though, such
as the case with V-MAX and Vorpal V2 disks. A few V-MAX V1/2 tracks
are actually over 8000 bytes long, like on Defender of the Crown.
-
Pete Rittwage
C64 Preservation Project
http://c64preservation.com
|
|
|
|
| Re: Serial line Burstnibbler, Was: [...] 0.4.0 of opencbm [...] [message #142364 is a reply to message #142328] |
Tue, 09 May 2006 18:10 |
Patryk 'Silver Dream
Messages: 737
Registered: July 2003
Karma: 0
|
Senior Member |
|
|
Wolfgang Moser wrote:
[...]
>
> Or with the MMC64.
Or with IDE64... ;-)
> I already heard some MMC64
> people asking for a Burstnibbler-alike tool,
> that does similar things as MNib. [...]
Huh, the stuff you discuss here is basically why I started the imaging
tool project. Since I can't really use mnib due to DOS/PC restrictions,
I wanted to have it on the 64. Currently I already have a working,
modular d64 imager, which uses GCR transfer but I would be very glad to
learn more about mnib's ways of creating the images as this was my
ultimate goal...
Will there be anybody who would be willing to give some explanations on
the typical mnib workflow/parameters?
P.S. The idea of transferring GCR with seven passes at first glance
seems to be absulutely ingenious...
|
|
|
|
| Re: New release 0.4.0 of opencbm (cbm4win/cbm4linux) [message #142365 is a reply to message #142358] |
Wed, 10 May 2006 04:38 |
|
|
Originally posted by: Dag Lem
Hello again, Wolfgang.
Wolfgang Moser <wnhp@d81.de.invalid> writes:
> Well with a disk recorded at 297rpm with a
> bitrate of 4/13 MHz and then played back at
> 303rpm, to which bitrate would this result
> then and what is the number of cycles passed
> away between two consecutive Byte Ready
> signals?
>
> Bitrate = ( 4 / 13 ) MHz * ( 303 / 297 )
> = 0.314 MHz = 314 kHz
>
> Cycles = 26 * ( 297 / 303 ) = 25.485
>
>
> This is not too critical, you only need to
> save one cycle. Aaaargh, one _more_ cycle ;-)
:-D
> Ok, now let's assume a tolerancy of +/-3%:
>
> Cycles = 26 * ( 290 / 310 ) = 24.322
>
> Well, tough, but I would believe that the PLL
> (not really a PLL) would not lock in at such
> a high discrepancy since some bitrates may be
> misinterpreted as other ones on the higher
> speed zones.
OK, so we're just guessing when it comes to the actual hardware limits ;-)
Another way of looking at this is to assume that the synchronization
is only limited by the actual bit patterns on disk.
If we limit ourselves to the GCR encoding scheme, we are looking at a
maximum of eight adjacent 1 bits or two adjacent 0 bits, and the
hardware syncs on 1 bits (see http://www.baltissen.org/htm/1541c.htm).
Thus, a maximum of ten consecutive bits are read without
synchronization (I am assuming that the synchronization is really done
on the transition from 0 to 1).
Sync markers may have more adjacent 1 bits, of course, but they are
not a problem here... unless they are exactly the number of bits
needed to trigger the sync hardware.. How many bits is that, by the
way? According to http://markus.brenner.de/mnib/vmaxtech.txt it's 12
bits, is this correct? I seem to remember 10 bits being mentioned
somewhere else. In any case, it is nothing to worry about when timing
read loops, since the sync marker is more than 8 bits and we thus have
time to spare.
Obviously, a bit cannot be reliably read if the read is offset by more
than 1/2 bit. I have no idea on the practical offset limit, but since
we're talking about worst case I'll assume an offset of 1/2 bit.
1/2 bit out of 10 bits amounts to 5%. 5% of 26 cycles amounts to 1.3
cycles, i.e. a byte can be read in 24.7 cycles, in theory.
So, spending 25 cycles on average in each iteration of a read loop
seems pretty safe.
> Any more questions? ;-))
Not at the moment ;-)
> You should ensure in any way that the V flag is
> cleared, _before_ the next Byte Ready signal is
> generated. Otherwise you would clear the flag for
> the just newly arrived drive. Since the following
> BVC * then waits again, you will have lost one
> byte.
Yes, that's what I thought, thanks. I believe the code examples I've
come up with should work. You just have to be careful about not
spending too many cycles between each bvc loop - in the concrete case
we're discussing the time spent between each read will be almost
constant (and thus has to be below 26 cycles). The only thing to look
out for is the time spent by loop instructions (e.g. dey / bne).
--
Dag Lem <resid@nimrod.no>
|
|
|
|
| Re: New release 0.4.0 of opencbm (cbm4win/cbm4linux) [message #142366 is a reply to message #142365] |
Wed, 10 May 2006 17:28 |
Pete Rittwage
Messages: 444
Registered: March 2005
Karma: 0
|
Senior Member |
|
|
Dag Lem wrote:
> Sync markers may have more adjacent 1 bits, of course, but they are
> not a problem here... unless they are exactly the number of bits
> needed to trigger the sync hardware.. How many bits is that, by the
> way? According to http://markus.brenner.de/mnib/vmaxtech.txt it's 12
> bits, is this correct? I seem to remember 10 bits being mentioned
> somewhere else. In any case, it is nothing to worry about when timing
> read loops, since the sync marker is more than 8 bits and we thus have
> time to spare.
It is 10, which is what V-MAX used on some later tracks. I'm prettty
sure the original 1540/1 schematics also mention 10 as when the sync
flag will be set.
We (or the hardware) 'lose' some 1 bits during long syncs also,
depending on motor speed. Technically, there is no "framing" during
this sync processing, so it can slip.
-
Pete Rittwage
C64 Preservation Project
http://c64preservation.com
|
|
|
|
| Re: New release 0.4.0 of opencbm (cbm4win/cbm4linux) [message #142367 is a reply to message #142365] |
Wed, 10 May 2006 18:05 |
Wolfgang Moser
Messages: 632
Registered: July 2003
Karma: 0
|
Senior Member |
|
|
Hello Dag,
you wrote:
> I wrote:
>> Ok, now let's assume a tolerancy of +/-3%:
>>
>> Cycles = 26 * ( 290 / 310 ) = 24.322
>
> OK, so we're just guessing when it comes to the actual hardware limits ;-)
That's right, _I_ don't know any documents with
"official" tolerancy specifications.
On another branch of this huge thread (wow!, when
did c.e.c see such an overwhelming heap of pure
technical stuff last?), Pete Rittwage mentioned
> Date: 09.05.2006 23:33
> Msg-ID: <44610AC5.8010409@rittwage.com>
> [...]
> A few V-MAX V1/2 tracks are actually over 8000
> bytes long, like on Defender of the Crown.
Ouch! Such a disk would have been written with a
bitrate of more than (5 Hz are 300 rpm):
8000 * 8 * 5 Hz = 320000 /s = 320 kbps
Hmmm, at which rotation speed should such a disk
been written with a fixed bitrate of 4/13 MHz?
300 rpm * (4000/13 kHz) / (320 kHz) = 288 rpm
The remaining question now is: Will such a
recorded disk work, when it is inserted into a
drive spinning at 312rpm???
I don't _believe_ so. I assume that the creators
of that copy protection (Defender of the Crown)
calculated with a drive tolerance of +/-1% as
with my earlier calculations. Doing the cycles
calculation under these assumptions results to:
Cycles = 26 * (288 / 303) = 24.71
And if a drive is adjusted optimally:
Cycles = 26 * (288 / 300) = 24.96
Way tooo critical, if you ask me. Was this copy
protection a reliable one?
> Another way of looking at this is to assume that the synchronization
> is only limited by the actual bit patterns on disk.
I can't follow you in this and do shorten out
some parts. I'm somewhat limited in time to dig
deeper and deeper into all of this ;-)
> [...]
> needed to trigger the sync hardware.. How many bits is that, by the
> way? According to http://markus.brenner.de/mnib/vmaxtech.txt it's 12
> bits, is this correct? I seem to remember 10 bits being mentioned
> somewhere else. In any case, it is nothing to worry about when timing
Well, read the schematics or better the "circuit
theory" chapters of the service manual of the
2031.
Yes, in fact, get the service manual of the
Commdore 2031 disk drive. The circuits we're
discussing here are the same for the 1541 (very
old, very long board ASSY #1540033) and the 2031.
But the schematics and service manual for the
2031 I know of are of much higher quality than
the ones for the 1541 (_I_ know of).
You maybe want to get the PDF converted version
from Rüdiger Schuldes (1.5 MiBiBytes ;-):
http://www.itsm.uni-stuttgart.de/staff/schuldes/online/serv2 031.pdf
See page 11, section "Read/Write Control Logic"
and the accompaying sheet #1 of the schematic
#2031038.
Look at U2L, 74LS133, the big NAND gate with
the 13 inputs. 3 of them are connected to
MODE (selecting Read/Write). 8 are connected
to the outputs of the shift register
(deserializer). Two more inputs are connected
to an "extension" of the shift register.
In sum, yes, 10 bits beeing all one in
combination with MODE beeing 1 (Read mode) are
needed to generate a SYNC signal at PB7 and
let counter U5L (74LS191, Byte Ready signal
generation) load its start value (resetting
byte framing).
> 1/2 bit out of 10 bits amounts to 5%. 5% of 26 cycles amounts to 1.3
> cycles, i.e. a byte can be read in 24.7 cycles, in theory.
>
> So, spending 25 cycles on average in each iteration of a read loop
> seems pretty safe.
This is an interesting point of view and an
interesting result, too.
Now, to get some practical experiences on how
much the bitrate can be "overdriven", we maybe
should ask Pete about the longest tracks he
ever saw in any copy protection. Pete?
Womo
|
|
|
|
| Re: New release 0.4.0 of opencbm (cbm4win/cbm4linux) [message #142368 is a reply to message #142367] |
Wed, 10 May 2006 18:53 |
|
|
Originally posted by: Dag Lem
Wolfgang Moser <wnhp@d81.de.invalid> writes:
[I wrote]
>> 1/2 bit out of 10 bits amounts to 5%. 5% of 26 cycles amounts to 1.3
>> cycles, i.e. a byte can be read in 24.7 cycles, in theory.
>> So, spending 25 cycles on average in each iteration of a read loop
>> seems pretty safe.
>
> This is an interesting point of view and an
> interesting result, too.
Note that this assumes the GCR scheme Commodore used. V-MAX used a
different scheme of groups of four bytes instead of five (at least
that's how I understand Harald Seeley's mention of "75% data, 25%
clock" opposed to Commodore's "80% data, 20% clock).
If V-MAX guarantees denser clocking (synchronization) than 10 bits, a
V-MAX byte can be read even faster than 24.7 cycles. If my theory
holds water, that is.
Assuming 8 bits, we get a possible speedup of 6.25% (0.5/8*100%),
i.e. a minimum time frame of 24.375 cycles (26*(1-0.0625)).
However, are the V-MAX bytes really recorded at maximum speed (26
cycles)?
Harald Seeley mentions 32 cycles a few times, perhaps all V-MAX tracks
are recorded at minimum speed? If this is the case, we have nothing to
worry about. Pete: Can you clear this up?
> Now, to get some practical experiences on how
> much the bitrate can be "overdriven", we maybe
> should ask Pete about the longest tracks he
> ever saw in any copy protection. Pete?
Yes, that would be the very interesting.
--
Dag Lem <resid@nimrod.no>
|
|
|
|
| Re: New release 0.4.0 of opencbm (cbm4win/cbm4linux) [message #142369 is a reply to message #142367] |
Wed, 10 May 2006 20:11 |
Pete Rittwage
Messages: 444
Registered: March 2005
Karma: 0
|
Senior Member |
|
|
Wolfgang Moser wrote:
> Now, to get some practical experiences on how
> much the bitrate can be "overdriven", we maybe
> should ask Pete about the longest tracks he
> ever saw in any copy protection. Pete?
In early V-MAX titles, such as the second Star Rank Boxing and early
Activision and Cinemaware implementations of V-MAX (we call it V2) there
are two tracks placed close together.
In this case, Star Rank Boxing, we have:
Track 16.0: 8024 (3) weak:230
Track 17.0: 8023 (3) weak:234
Now, the weak bits are at the end of the track (or the beginning, it's
all relative) but they are detectable in the same place at each
rotation, so the track really *is* that long, plus or minus a few bits
for slippage.
I've never tried to slow to 288 to write these V-MAX tracks before, I've
just been able to 'compress' the data on the track (basically filter
unneeded long syncs and gaps) and get them to remaster.
The latter Vorpal disks have tracks a little over 7800 bytes at density
3, and the motor must be slowed to <297 to write those, but they work
when that is done. They lack syncs, as we've discussed, so no
compression or filtering could help us remaster these.
-
Pete Rittwage
C64 Preservation Project
http://c64preservation.com
|
|
|
|
| Re: New release 0.4.0 of opencbm (cbm4win/cbm4linux) [message #142370 is a reply to message #142367] |
Wed, 10 May 2006 20:12 |
Pete Rittwage
Messages: 444
Registered: March 2005
Karma: 0
|
Senior Member |
|
|
Wolfgang Moser wrote:
> Now, to get some practical experiences on how
> much the bitrate can be "overdriven", we maybe
> should ask Pete about the longest tracks he
> ever saw in any copy protection. Pete?
In early V-MAX titles, such as the second Star Rank Boxing and early
Activision and Cinemaware implementations of V-MAX (we call it V2) there
are two tracks placed close together.
In this case, Star Rank Boxing, we have:
Track 16.0: 8024 (3) weak:230
Track 17.0: 8023 (3) weak:234
Now, the weak bits are at the end of the track (or the beginning, it's
all relative) but they are detectable in the same place at each
rotation, so the track really *is* that long, plus or minus a few bits
for slippage.
I've never tried to slow to 288 to write these V-MAX tracks before, I've
just been able to 'compress' the data on the track (basically filter
unneeded long syncs and gaps) and get them to remaster.
The latter Vorpal disks have tracks a little over 7800 bytes at density
3, and the motor must be slowed to <297 to write those, but they work
when that is done. They lack syncs, as we've discussed, so no
compression or filtering could help us remaster these.
-
Pete Rittwage
C64 Preservation Project
http://c64preservation.com
|
|
|
|
Members
Search
Help
Register
Login
Home
