File reading, large numbers and database functionality

[SBRuss]

Hi all,

I'm a new Smart Basic user, just starting to look at my first program. I'm more interested in productivity and data driven applications than graphic action games, so data manipulation functionality is important to me.

I've had a read through the FILE functions, which seem the base I'd have to build from to make something along the lines of SQL functionality. I have a couple of simple questions about file reading and wriring to start.

Q1. The FILE N$ WRITE X function only writes a single byte. What's the suggestion for storing numbers > 255? Do they need to be stored as a string, and then converted back to a number when read in? The automatic string to number conversion should make that easy, but I'm not sure of the performance overhead if it is being done frequently.

Q2. Do the FILE - READ/WRITE functions move the file pointer, or any of the read & write functions? The manual is a little unclear. It states on some functions that they don't affect the pointer, but does that mean any that don't state it, will move the pointer? If no, that means to process through the file byte by byte requires using the SETPOS command every time?

Thanks.

[Mr. Kibernetik]

Q1. The FILE N$ WRITE X function only writes a single byte. What's the suggestion for storing numbers > 255? Do they need to be stored as a string, and then converted back to a number when read in? The automatic string to number conversion should make that easy, but I'm not sure of the performance overhead if it is being done frequently.

FILE READ and FILE WRITE commands work with single bytes only.
Smart BASIC does not store numbers as binaries. If you need to save numbers to file you can write them as text using FILE PRINT and FILE INPUT commands.

It states on some functions that they don't affect the pointer, but does that mean any that don't state it, will move the pointer?

Yes, it does. If it is not stated that file pointer is not used then it is used.
File commands functionality is explained in the beginning of "Files" section of the manual. There are two groups of files commands. Commands of 1st, simple group does not use file pointer. Commands of 2nd, advanced group use file pointers.
File pointer changes its position automatically, you do not need to change it manually after each file command.

[SBRuss]

Thanks for the clarification.

Unfortunately FILE PRINT and FILE INPUT are not sufficient for my needs. I want to have the granularity to read & update only one field (which could be 1 or more bytes) at a time, not have to rewrite the entire file for each change. I guess I could store everything in a memory structure and then batch save it at some later time, but this could mean a large memory footprint for the program, and the risk of losing data if not backed up regularly. Maybe what I want to do isn't possible due to either memory or file i/o overheads. I guess I'll just have to do some experimenting and see what works.

[Mr. Kibernetik]

Thanks for the clarification.

Unfortunately FILE PRINT and FILE INPUT are not sufficient for my needs. I want to have the granularity to read & update only one field (which could be 1 or more bytes) at a time, not have to rewrite the entire file for each change. I guess I could store everything in a memory structure and then batch save it at some later time, but this could mean a large memory footprint for the program, and the risk of losing data if not backed up regularly. Maybe what I want to do isn't possible due to either memory or file i/o overheads. I guess I'll just have to do some experimenting and see what works.

If you see some limitations in sB and have suggestions how it can be improved, your suggestions for improvement are velcome!

[SBRuss]

OK, will do. I do need some more experience though before I have a good idea.

[Henko]

Thanks for the clarification.

Unfortunately FILE PRINT and FILE INPUT are not sufficient for my needs. I want to have the granularity to read & update only one field (which could be 1 or more bytes) at a time, not have to rewrite the entire file for each change. I guess I could store everything in a memory structure and then batch save it at some later time, but this could mean a large memory footprint for the program, and the risk of losing data if not backed up regularly. Maybe what I want to do isn't possible due to either memory or file i/o overheads. I guess I'll just have to do some experimenting and see what works.

Hi,
I fully understand what you mean.

For large amounts of data i would also prefer a permanent database using the FILE mechanism, instead of loading an entire file, do the querying and updating, and saving the lot at the end of processing.

In SmartBasic, i would develop a database system along the following lines:
1. A "database" file with fixed lenght "slots" (records), enabling calculation of the filepointer for a desired slot, and setting it using the SETPOS command.
2. A user function to "pack" the elements of the record lay-out into one string, and a function to "unpack" a string into the individual data elements (the SPLIT command is very usefull here). I would the use the READLINE and WRITELINE commands to retrieve and write records.
3. The maximum length of records must be known and taken as the fixed length of the database "slots".
4. A user function which forms a key for each record. This may be one of the data elements, or a combination of data elements, depending on the query needs. Those keys could be kept in a separate file for processing speed, but thie file should be maintained in parallel with the database.
5. A hashing algorithm and -function to calculate a slotnumber for any given key. Together with the fixed slotlength the filepointer can easily bet set and the database record retrieved or written.
6. The adress calculation will result in the same slotnumber for different keys in a number of cases ("synonyms"). For this reason, the database must be split into a "primary" part and a "secondary" part which functions as an overflow area for the "synonyms". Synonyms have an entry point in the primary area (the first record), and a linked list of the other records in the secondary area. In extremum, if one defines the size of the primary area as one (1) slot, the entire database is one linked list, residing in the secondary area (except the first record).
7. Of course functions must be written for querying, adding, modifying and deleting records. Additional functions might be developed for keyfile (re)building, reports, conversion of existing files.
8. This system is relatively simple for databases with one, predefined key. For SQL like possibilities, a flexible key selection mechanism must be developed. I did not think that over yet.

I already wrote a set of functions to deal with the hashing calculation, the partioning of the database and the linked list handling of synonyms (not for using it, but for the fun of programming ). I can upload it here if you want.

Looking forward to your SQL DBMS

[SBRuss]

That's quite some requirements

The main limitation I see there is the fixed size slots. I guess smart data structure design might be anle to get close to that. I'm sure I must have done something like this when I was writing COBOL about 25 years ago. Unfortunately, that code is long lost, just like my memory of thise says

[Henko]

That's quite some requirements

The main limitation I see there is the fixed size slots. I guess smart data structure design might be anle to get close to that. I'm sure I must have done something like this when I was writing COBOL about 25 years ago. Unfortunately, that code is long lost, just like my memory of thise says

Well, i have no requirement. I reacted on your saying :
"I've had a read through the FILE functions, which seem the base I'd have to build from to make something along the lines of SQL functionality. I have a couple of simple questions about file reading and wriring to start."

The fixed size slots need not be a problem if the data do not vary to much in length per record. There will be some spillage, but who cares in the era of terabytes?

[SBRuss]

Well, i have no requirement. I reacted on your saying :

Yes, I just meant the rquirement to do what I need. Thanks for the suggestions.