KSEQ to ESEQ file conversion

[jonrmorgan]

Hi everyone,

I'm working on the next version of my SY Manager program - SYM3 will go to beta later this year.

One of the most requested features I've been asked for in this version is to provide a way of extracting voice and other data - particularly sequences - from floppy disk files in the various Yamaha proprietary formats (.t01,k01,l01 etc.)

The good news is I have reverse engineered the file formats for the synthesiser element of these files and SYM3 will extract SY77 or SY99 voice, multi, microtuning, pan and system setup data. It will also be able to edit all these patch types.

I'm now trying to decide what to do with sequencer data in floppy files. I had planned to convert ESEQ format data to standard Midi file format 0. Unfortunately it's clear that .t01 files - the most commonly used - store the sequencer data in KSEQ format. This is literally a real headache because I can't see any obvious correspondences in the byte data of these formats.

I am aware that dear old Yam made available an ESEQ to MID file converter and that does indeed work fine on SY77/SY99 ESEQ( .l01 ) files. But I really need a KSEQ file format map as otherwise users who are hoping to extract sequences from .t01 files are going to be disappointed.

So any assistance with cracking these formats would be much appreciated.

Jon

http://www.fm-alive.com

[Clyde]

Hi Jon,
Good to see you here again! I don't have any help for you, just want to offer my appreciation for what you have done with your programs and website. What you have done has helped to broaden my SY77/99 experiences the last few years.
Clyde

[Derek]

Hi, Jon

I now do a Java based librarian for the SY77 (along with ones for my other synths), and I've decoded the KSEQ format down to track level (so I don't know how the MIDI data in tracks is currently structured).

If you're happy to share info on the SY99 file format differences (compared to the SY77) if you already have those figured (including user Waves), I'm happy to trade this KSEQ info. That will save us both some effort.

I'm now away for a few days, so will be out of the loop until the weekend, but if you're interested then please send me a PM.

[jonrmorgan]

Thanks Clyde - let's see what comes up. My feeling is that someone at Yam ought to be able to provide this as ESEQ at least is still widely used in their equipment (even mobile phone chips I believe !). I can't believe it's a proprietary issue as the standard MIDI file took over years ago.

Just looking at the byte streams my guess is that KSEQ is a 7bit equivalent of ESEQ.

I have tried to find someone at Yamaha that has the knowledge several times over the years and eventually got an apologetic response from a suit in Japan to the effect that there was now no-one in the company that could help. Perhaps there's someone on the Disklavier side but it's an extremely opaque organisation.

The only other potential source I've come across is the mysterious Giebler Enterprises but they don't respond to mail as I see from some of the postings on this site.

Jon

http://www.fm-alive.com

[bnz99]

Hi guys, I know I'm kind of late to the party here (but I think that kind of happens a lot here as far as I have seen). I have been looking into what I can do with the song data of my SY77 and came across a SYKSEQ file format specification: http://www.mediafire.com/file/njjjyrqzd ... Y.pdf/file. I was brave enough to click this kind of obscure link and actually was kind of surprised that there is actually something in there that might even make sense.

[jonrmorgan]

I guess a 13 year lag is a bit of a stretch ! But I'm still interested in a solution to this problem. I did see this file back in 2010 and whilst it is helpful on the file header structure, the information on track data is just not sufficient to code a converter. I did try to contact the author, Pekos Bill (really), without success. As Derek said without the track data format we're stuck. Thanks anyway.

[animaux]

If you don’t mind another stretch of only three years this time … I’ve been interested in converting KSEQ directly to MIDI for some time since a friend of mine has a large archive of compositions.

I’ve been looking into the very helpful files left behind by the mysterious »Pekos Bill« and also the manual of the Yamaha v50 manual, which offers a lot of details about the NSEQ-Format, which seems to be a kind of missing link between ESEQ and KSEQ.

So this has been some kind of rosetta stone for me and I think I figured out the note structure of the KSEQ files. (I’ve not looked into the pattern allocation table. I think Derek figured that part out, since his sy.factory is able to show some info about these.)

I have built a tool to analyze KSEQ Files and build a MIDI file in Javascript. It runs as a website. The main problem I am facing is that different tracks of a KSEQ seem to have different temporal resolutions in either note lengths or delta times.

I have a KSEQ file that I also have as a MIDI file recorded form the SY77 to a sequencer, so I can compare the resulting files note by note. So there is this one track that fits perfectly, but the others don’t. It might also have something to do with quantisation in the SY sequencer, that may affect the temporal resolution of certain tracks? The other problem with this track is, that at some point the timing of the otherwise working track is suddenly off too. That might as well be a problem with my converter I haven’t been able to iron out yet.

So, the note structure has some similarities with NSEQ, but differs a lot too. It has short and long notes that include the length (no note-offs as in MIDI). Also there are short and long times, or deltas, comparable to MIDI but encoded a bit differently:

EDIT, this has to be revised as it’s not totally right —> see revised format here: viewtopic.php?p=127385#p127385

I’m happy to PM anyone a link to my analyser/converter, if you want to give it a try. Though like I said it is not working properly yet.

There are some bits that »Pekos Bill« hadn’t figured out yet. Of particular interest could be the the unidentified bits in the sequencer song data region after COM-KSEQSY1_SEQ —> 0x40F to 0x421 (with the tempo in between).

All the best,

Alexander

[Derek]

Interesting what you have found. I never really have to the time or inclination to look at this, as I do not use internal sequencers and could never fathom it out, so it is good that somebody is looking at it

And interesting what you found in a manual for the V50 - who would have thought of looking there?

Good luck!

[animaux]

Thank you for commenting on this, Derek!

I totally understand you don’t want to invest time into this, but I was hoping that you might have some info on the header part of the sequencer song I mentioned above (0x40F to 0x421), since you obviously took some effort to decode the info on Tracks, Patterns, User Pattern Chains and User Time Signatures.

Actually it was also »Pekos Bill« who mentioned that the NSEQ format was documented in the V50 manual. Sadly Yamaha didn’t carry on with such documentation in later synths.

[bnz99]

I'm interested in giving this a test. I have some sequences where i always wanted to see the midi data of.

[jonrmorgan]

Well, good things come to those who wait ! Even though it's 16 years since my original post I'm sure a KSEQ to MIDI converter is still needed.

Thanks for working on this - to help you debug your app I thought you might find it helpful to have one of the original demo disk songs (Arothewo) in KSEQ,ESEQ,NSEQ and Standard MIDI file format 0 for byte level comparison. I downloaded the files from my SY99 and SY77 (NSEQ), zipped them up and put them on my One Drive : https://1drv.ms/u/c/0a5e9480fa3c6d95/EY ... g?e=Nt56nc

It looks like you got a good deal further than I did in 2006 - the Pekos Bill doc was helpful but has frustrating gaps.
Good luck - let me know if I can help further with your project

Jon

[animaux]

Thanks for your replies, especially for the demo song, I’m keen to try how that works out with my converter. The File with .Mxx-extension will not work straight away with my converter, since it awaits a file with a .Kxx extension that has the SY1_SEQALL identifier with the sequencer settings, followed by COM-KSEQSY1_SEQ later on, so all the offsets will be off. I can probably build the converter to open those files as well of that, but not for now. I can simply add a set of SY1_SEQALL by hand to test this out.

I’m outputting to MIDI file format 1, maybe that’s part of the problem? Though it shouldn’t.

Also I’ve only tested it with sequences from the SY77. I don’t know if there are any differences in the KSEQ files.

I’ll just leave the link to my converter here, actually there is probably no need to only send it by PM. It comes with an MIT license, so no warranties But I’m of course interested in fixing it, so I’m happy for any helpful feedback:

https://animaux.de/kseq/

[animaux]

Thanks again Jon, I padded the KSEQ.M01 with the SY1_SEQALL-section of another file and of course there are some bugs with my code. Since I haven’t come across any aftertouch pressure- and pitch-bend-events in the sequences available to me so far. The event list will show them, but no MIDI file will be created. Will try to fix this just now.

***Update***

It now runs again and should process the mentioned controller events properly.

Looks like the same issues with timing are coming up when converting the demo song. But this will hopefully help me to find the problem. Also the 5/4 time signature is not taken into account yet.

Comparison of the demo song MIDI with the KSEQ converted to MIDI in Logic.

[animaux]

Since I don’t own a SY myself. Would any of you be able to record a sequence for me, with the following tracks?

1. Track: Full notes
2. Track: Half notes
3. Track: Quarter notes
4. Track: Eight notes

It can be all C-Notes, or whatever, only the timing matters.

It would help even more if this was quantized to ⅛. Though I have some suspicion that quantizing turned on might affect the temporal resolution of a track.

And if you can stand playing this dull sequence long enough, a full minute or more would also be helpful. Maybe is is possible to copy/paste? But it shouldn’t be in pattern mode.

Thanks in advance for any input!

[bnz99]

Here are some kseqs in song mode. Each kseq file is one bar of C notes quantized. I have quantized full and half note to 1/2, the quarter to 1/4 and eitgth notes to 1/8. If I should repeat this and quantize all to 1/8, let me know. About that kseq with a full minute: what notes should this be? or should the vary full notes, half notes, quarter notes per bar or something like this? I hope the access to dropbox works. I think they may have put an email registration in front of it.

https://www.dropbox.com/scl/fo/ed3qxkpw ... tfmvb&dl=0