Talk:Nitrofs

Revision as of 04:03, 22 June 2009

Changes notes

Just today (2008-05-01) heard there was a discussion about nitrofs on gbadev's forums, and found out why i was not able before to use a .nds file with emulators. Both thanks to Wintermutes Thx!!!!!! :DD

After reading the forums found out there was a bug with fseek() and the SEEK_CUR, i am sorry. u_u This was not noticed immediately as I only used SEEK_SET in testing. The problem turned out to be an improper return value in my fseek() function. Guess i should thank noda for pointing this out, and btw, hellos! ^^/

Now, a single .nds file should work equally well with both emulators and flashcards, hopefully... I tried this before in the previous version but did not realize passing -o option to ndstool prevented GBA access. After fixing the Makefile and zeroing the offset used previously for the older style .ds.gba

2008-05-21 - fixed ftell() issues, and re-enabled support for ds.gba files in addition to .nds on emulators (best of both worlds! :PP).. also added stat() support for reading file lengths w/o opening..

2008-05-26 - Added chdir() functionality, lets see how long till someone usurp the insanely simple method...

2008-06-16 - Relased turbo version.

2008-06-19 - So soon another release :D this time "." and ".." is fully supported!! YAY!!

Why not read from the card directly?

Noticed someone asked about this on another forum. I'll try to explain (based on what i've been told so please correct me if am wrong)...

Original games are encrypted and read over SPI buss from the card which decrypts using hardware. When these games are loaded off flashcards, the flashcard's loader modifies the card/SPI functions which are patched to use flashcard specific functions that emulate normal SPI access to the card's data.

The details of this are foggy, since flashcard companies don't wanna say what they are doing exactly and help competitors. :/ So to make homebrew that uses official SPI style calls would be sort of pointless since the loaders would prolly not pick up on this, not patch them, and thus not work at all.

eris ♥

Multiple vs Single FAT filehandles when in .nds mode

After pondering this a bit, have decided most uses would prefer quicker open/close times versus a very slight slowdown when reading from multiple files.

When using the .nds file and FAT, is there any advantage in using opening only one ".nds" instead of opening a ".nds" file for each file opened on the nitrofs.

fopen uses up quite a few cpu cycles, so obviously f/opening of files from the nitrofs would be faster if the same .nds file were used instead of opening a new one each time. This would speed up most other operations as well such as directory openings, file/dir closes, stat() calls, etc.

What happens if your reading from two nitrofs files at once?? In this case the single .nds file would need be seeked back and forth like crazy every time a different nitrofs file was read from. Have heard that FAT may take a bit of time to search the allocation tables and change file position, is this take any longer than opening up multiple files that hafta read from different positions on the card, and how the different DLDI implementations would affect this is truly a guess to me??? If anyone wants to try I'd like some feedback here.

I tested it with my newflash 8gbit, here are the results:

• all results in ms

2files
this test reading 2 files simultainiously, 0x100 bytes at a time, over 10327 reads (file is 1032793 bytes), one file is fopen'd other is open'd.

As i suspected, reading multiple files with single .nds file is slower, but.... not by much at all, especially given the number of reads this spans.

fread & read
just freading (or reading) the same 1032793 byte file. Times include open/close calls. here the winner is the single file mode, saving about 106ms for both methods. open is always gonna be a bit faster because its a simpler method than fopen (this is why i prefer open to fopen *shrug*)

shortread
ok this is reading a short 16 bytes or so from a file and doing a bit of seeking around. Here the biggest advantage can be seen. If your opening alot of small files (as i suspect most will be doing) there would be a big improvement as all those fopen/open calls add up.

Bug in ndstool

Screenshot showing original file included into the nitrofs on the left, and the ndstool generated .nds file with mangled last byte on the right.

There is a bug in ndstool causing the last byte of the last file included into nitrofs to be 0!! Its a wonder no one has spotted this before rly. Thanks to Lacey for pointing this out and helping to find/fix it.

The bug was in the way it ensured the filesize would align along 4 byte borders, always zeroed last byte of file. Below is a diff, and a precompiled version until the next DKP update.

If you use the binarys or recompile the cvs sources, it may say something like:

Cannot open file 'c:/devkitPro/libnds/default.arm7'.

thats because the name of the default arm file has changed from basic.arm7 to default.arm7, just copy or ln -s basic.arm7 to default.arm7 and it should work. :D

• precompiled linux version

heres a diff:

Index: ndstool/source/ndscreate.cpp
===================================================================
RCS file: /cvsroot/devkitpro/tools/nds/ndstool/source/ndscreate.cpp,v
retrieving revision 1.44
diff -u -r1.44 ndscreate.cpp
--- ndstool/source/ndscreate.cpp        14 Apr 2008 02:31:14 -0000      1.44
+++ ndstool/source/ndscreate.cpp        19 Jun 2008 04:12:45 -0000
@@ -581,10 +581,12 @@
 
        // align file size
        unsigned int newfilesize = file_end;    //ftell(fNDS);
-       newfilesize = (newfilesize + 3) &~ 3;   // align to 4 bytes
-       header.application_end_offset = newfilesize;
-       fseek(fNDS, newfilesize-1, SEEK_SET); int c = fgetc(fNDS);
-       fseek(fNDS, newfilesize-1, SEEK_SET); fputc((c >= 0) ? c : 0, fNDS);
+       if(newfilesize & 3) {
+               newfilesize = (newfilesize + 3) &~ 3;   // align to 4 bytes
+               header.application_end_offset = newfilesize;
+               fseek(fNDS, newfilesize-1, SEEK_SET); int c = fgetc(fNDS);
+               fseek(fNDS, newfilesize-1, SEEK_SET); fputc((c >= 0) ? c : 0, fNDS);
+       }
 
        // calculate device capacity
        newfilesize |= newfilesize >> 16; newfilesize |= newfilesize >> 8;

Games/Projects using Nitrofs

Below is a list of projects using the nitrofs driver. Thought of adding this to main page but figured should start here first. Curious who is using this and what they're working on, add to list if you wish...

• Thds01 - Touhou project for nds (this is what the library was originally created for :)
• Lacy's Investimigations
• Indiana Jones and the Aztec Curse

Thank you :D

Finding filename for nitroFSInit()

Here is some code I wrote to find the filename of the currently loaded NDS file. It relies on the fact that the header is quite probably unique for any given ROM, and so scans through the filesystem and checks all NDS files. It seems to run quite fast enough to be useful.

int NDSFilePathScanner(char *path,char *end,void *buffer)
{
	static struct stat st;
	DIR_ITER *dir=diropen(path);
	if(!dir) return 0;
 
	while(dirnext(dir,end,&st)==0)
	{
		if(end[0]=='.') continue; // skip all entries beginning with a .
		else if(st.st_mode&S_IFDIR)
		{
			char *newend=end+strlen(end);
			newend[0]='/';
			newend[1]=0;
			if(NDSFilePathScanner(buffer,newend+1,buffer)) return 1;
		}
		else
		{
			char *ext=strrchr(end,'.');
			if(ext&&!strcasecmp(ext,".nds"))
			{
				int fd=open(path,O_RDONLY);
				if(fd>=0)
				{
					if(read(fd,buffer,0x170)==0x170)
					if(!memcmp(buffer,(void *)0x27ffe00,0x170)) return 1;
					close(fd);
				}
			}
		}
	}
	dirclose(dir);
	return 0;
}
 
char *FindPathToCurrentNDSFile()
{
	static char path[MAXPATHLEN];
	static uint8 buffer[0x170];
 
	path[0]='/';
	path[1]=0;
 
	if(NDSFilePathScanner(path,path+1,buffer)) return path;
	else return NULL;
}

Feel free to add this to nitro.c too. I'd suggest making an init function that takes argc, argv and a default filename, and uses the name supplied by argc and argv if available, else tries to use this code to find it, and if that fails, uses the default name passed in. That is what my code does at the moment, anyway. -- WAHa.06x36

* Oh, thanks.

been wanting something like this but too busy to write it.. ill give it a try later, might tweak a few things, but otherwise looks very promising.

http://nocash.emubase.de/gbatek.htm#dscartridgeheader <-- Ohh i didn't know this! Was wondering where 0x27ffe00 came from. Very nice :PPP

Well tried it...

Hmmmmmz, after spending all night on this... a few issues:

• open()'s filename NEEDS "fat:/" at the start.... took a while but figured that one @_@
• my card's loader changes the header!?!? here may be real problems... u_u;; about 8 bytes starting at 0x60 are different

 060h    4     Port 40001A4h setting for normal commands (usually 00586000h)
 064h    4     Port 40001A4h setting for KEY1 commands   (usually 001808F8h)

may be able to skip these and continue.. using ndsloader.nds it works fine, tho ndsloader also sets arg0 properly so idk...

yes comparing 0 to 0x60 then 0x70 to 0x170 was successful, finally! :D Tho im wondering about other cards??

Summary so far

anonuser figured out that cartridge header in .nds is copied to 0x27ffe00 during loading and is mostly unique to each file/build. By comparing the first 0x170 bytes starting with 0x27ffe00 at startup with each file on the FAT filesystem it is possible to locate the .nds file. Great idea until the loaders are standardized to provide proper argv argc values. :D

there is a potential problem with card's loaders modifying the headers... however this may have been fixed.(see above)

The code provided by anonuser was modified a bit, and the nitroFsInitAdv() function was added. It workes as follows:

first argument fatfilename is the name we expect the file to be, if this is incorrect the function will return pointer to what file it found, if correct will return fatfilename, or NULL on error. Also it will return fatfilename if it is in GBA mode.

First checks if its in GBA mode, if so will not call fatInitDefault(), and returns fatfilename. In this way there is no startup delay on emulators. :D If not GBA first fatInitDefault is called, then a stat of fatfilename is done to detemine if the file exists, if so, it attempts to use this. if not will search thru all files on the card ending in ".nds" for a matching header.

Proof of concept. Code needs cleanup work but atm it needs testing on several cards:

bool NDSFilePathScanner(char *path, char *end, void *buffer, struct stat st)
{
	DIR_ITER *dir=diropen(path);
	if(!dir) return false;
 
	while(dirnext(dir,end,&st)==0)
	{
		if(end[0]=='.') continue; // skip all entries beginning with a .
		else if(st.st_mode&S_IFDIR)
		{
			char *newend=end+strlen(end);
			newend[0]='/';
			newend[1]=0;
			if(NDSFilePathScanner(path,newend+1,buffer,st)) {
				dirclose(dir);
				return true;	
			}
		}
		else
		{
			char *ext=strrchr(end,'.');
			if(ext&&!strcasecmp(ext,".nds"))
			{
				int fd=open(path,O_RDONLY);
				if(fd>=0)
				{
					int len;
					len=read(fd,buffer,DSHEADERLEN);
					if(len==DSHEADERLEN) {
						if((memcmp(buffer,(void *)DSHEADERCPY,0x60)==0) &&
							(memcmp(buffer+0x70,(void *)DSHEADERCPY+0x70,DSHEADERLEN-0x70)==0)) {
							close(fd);
							dirclose(dir);
//printf("FOUNDIT!\n");
							return true;
						}
					}
					close(fd);
				} 
			}
		}
	}
	dirclose(dir);
	return false;
}
 
#define FATPATH "fat:/"
 
//path should be char path[MAXPATHLEN];
//returns NULL if failed
char *findPathToCurrentNDSFile(char *path, struct stat st)
{
 
	uint8 buffer[DSHEADERLEN];
	strcpy(path,FATPATH);
//	path[0]='/';
//	path[1]=0;
 
	if(NDSFilePathScanner(path,path+strlen(FATPATH),buffer,st)!=false)
		return path;
	else
		return NULL;
}
 
//quick hack to simplify nitrofs usage
// * fatfilename - what the developer thinks the file is gonna be called on the fat filesystem.. 
// return value is == submitted fatfilename if its correct or is using GBA mode. 
//  if fat is detected but the fatfilename fails to open, or fatfilename == NULL, then it attempts to find 
//  the correct fatfilename by searching thru all of fat. If found will return pointer to the newpath, if not
//  will return NULL indicating no file was found. by comparing the return value to fatfilename devel can 
//  deterimine if the location of file has changed, this value may then be stored in nvram and reused next time! ^^
const char *nitroFSInitAdv(const char *fatfilename) {
	static char path[MAXPATHLEN+1]; //+1 for the zero that is sometimes added in these functions
 	struct stat st;
//These two lines to detect GBA mode should be put into a function in nitrofs, the rest stays here...
	REG_EXMEMCNT &= ~ARM7_OWNS_CARD; //give us gba slot ownership
	if(strncmp(((const char *)GBAROM)+LOADERSTROFFSET,LOADERSTR,strlen(LOADERSTR))!=0) {
//printf("yes its fat!\n");
		if(fatInitDefault()==true) {
			if((fatfilename==NULL) || (stat(fatfilename,&st)==-1)) {	//no filename set
//printf("searching for the path\n");
				char *foundpath;
				foundpath=findPathToCurrentNDSFile(path, st);
//printf("foundpath=%s\n",foundpath);
				if(foundpath) {
//printf("found '%s'\n",path);
					if(nitroFSInit(path)==0) {
						printf("nitrofail?!\n");
					}
				}
				return(path);
			} else {
//printf("using existing path\n");
				nitroFSInit(fatfilename);
				return(fatfilename);
			}
		} else {
			printf("Fat failed!\n");
			return(NULL);
		}
	} else {
//printf("using GBA mode\n");
		nitroFSInit(fatfilename);	//actually all these should be checked ~_~
		return(fatfilename);
	}
}

A working nds for testing on different cards is found here: nitrofs.nds Will say "This file is being read off nitrofs!! :DD" if successfull. Should be able to put this in any folder on the card, and it still finds it!

List of tested cards/loaders

• neo 8gbit - works
• ndsloader.nds - works
• moonshell - works

(version numbers would be nice ~_~)

Eris

I've tested it with the folowing cards:

• r4ds v1.18 - diffs at 0x18 - 0x1a, works when ignoring 0x10 - 0x20
• acekard2 akmenu 4.16 hw:80 - diffs at 0x160 - 0x167, works when ignring 0x160 - 0x170

Tiger[DS]

Some comments

Looking good so far. I wonder if it's wise to just use the default filename if it exists, though. Perhaps the code should read the header first to check that it is indeed the right file? If it isn't, it could run a scan, and if that fails too it could use the default filename anyway (for cards that change some more header fields). I can imagine, say, that you've got two different versions of an app, and put one in the root and one is a subdirectory. That would break with the current code.

• yeah considered that, needs to actually check the header still just in case that one in a million chance. My intention is that fatfilename be stored in nvram. First time the game is ran this should be set to default values prolly "" or all zeros. Then the user checks afterward if the returned value is different, if so they store the returned value in nvram for next time. Using that scenario really does seem difficult to mess up unless, as you said they for some reason have same .nds in two directories, but even then, if they are identical then the nitrofs data would be identical and game would run normally. Gah its confusing @_@ bring on argv!! Also regarding argv, in the up comming libnds release there is a magical number that indicates argv and argc are set to proper values. may be able to work this into the scheme somehow, although if your using the new libnds, then you prolly arn't gonna want/need/or even really be able to use this anyway because the nitrofs driver in new libnds does not take any arguments to nitroFSInit().. *headspin* Eris

Also, maybe the code should take argc and argv as parameters too, so the user doesn't need to check those for themselves? Just for convenience.

Also, detecting GBA mode is a really good idea, because goddamn, libfat sure takes its time to figure out that it doesn't work on no$gba.

• yeah, got the idea because wintermute had made nitroFSInit() go ahead and init fat when adding the code into upcomming libnds release. However then we're stuck with: How do you determine if fat was init or not??? because the user may wanna use fat seperately.. (on some rare ancient flashcards it will load into gba mode, but still use fat @_@) This is why we should prolly make the GBA mode detection function publicly available from nitrofs itself. Eris
  • I just noticed the GBA mode detection fails on Ensata. It tries to init FAT, which fails. I'm not sure yet what causes that, I'll look into it later. -- WAHa.06x36
    • heys, sorry i havent gotten back on this yesterday a server was compromised, and been working like mad to get it secured backing up data, worrying alot, etc... hopefully wikidata will be preserved... >_< should get back on this by the weekend. Also more info on ensata would be nice, i imagine they also screw with the header somehow. might want to do a hexdump of the gba area see if PASS is readable.. thx! Eris

Are you going to be putting this into the official nitrofs.c file? I'd love to cut down the clutter in my own code by offloading it onto you.

• have put it into a seperate ndsfinder.c and .h file. Just because its a temporary measure until the new libnds comes out, and would save a few hundred bytes to omit it. In nitrofs.c I may add a isGbaMode() function because this may be useful to devels who would like to know if FAT was initialized or not. I'll get back on this in a bit, and upload ndsfinder.c tho its pretty much just the code i pasted so far. ^^

--WAHa.06x36

Hey, you still around? been busy fixing my server and beefing up security. Also had to put some stuff on ebay to help pay for the server, and of course working on my Touhou for DS game quite alot. Curious if you have made any progess here?

Eris