# =========== Header =========== # File: Storage on NewtonOS # Project: Newton Bowels # Written by: Paul Guyot (pguyot@kallisys.net) # # Created on: 07/11/2000 # Internal version: 3 # # Copyright: © 2000 by Paul Guyot. # All rights reserved worldwide. # =========== # =========== Change History =========== # 12/08/2000 v3 [PG] Fixed some typos # 12/05/2000 v2 [PG] More informations about the TStore interface # 07/11/2000 v1 [PG] Creation of the file # =========== Introduction ============ This document explains the low-level functionning of the storage on NewtonOS. The system uses persistent transactional object stores to store the data. Here, we analyze the interface to these store objects, TStore. You should be familiar with Apple's documentation concerning storage on NewtonOS. This is a complement to what Philz discovered ===== Exercpt from Philz' page http://vyx.net/~philz/hammering.php3 ===== Flash store Chips MP130 flash store is made of 2 chips: Ä AMD AM29F040 - 512KB Ä Intel E28F008SA - 1MB Both chips have 64KB write/erase blocks. They are accesible from 0x20000000. I don't know why, but reading from longs from memory gives us <0,stream1,stream2,garbage>. Couldn't it be just mapped 1MB of Intel and after that 512KB from AMD ?????! Stream 1 contains AMD data, and stream 2 data from Intel flash. Total MMU mapped flash area covers 8MB. AMD is repeated 4 times and Intel 2 times in their streams. Blocks Both streams are divided into 64KB blocks. Each block structure looks like From To What 0x00004 0x0fc00 Objects 0x0fc00 0x10000 Block information Some objects are "local" to block they are in. Their flags is 0xfffe, and OID is 3. In every block is at least one object like that, it is from 0x4 to 0x810. Sometimes there is more objects like this, but they have size of 64 bytes. Ä Tool for extracting objects from your Newton[2] (dir, tar.gz) Package Store Package stores keep objects just like any other stores. But their structure is much simpler, because they keep read-only data. Package soup stream looks that: 0x00000000 number of objects offset to object 0 offset to object 1 ... offset to object n size of store After that goes unaligned object data. Ä Tool for extracting objects from package stores[1] (dir, tar.gz) THIS SECTION IS COMPLETE :) ===== End of the Excerpt ===== TStore is the protocol interface for stores. There are three implementations, TMuxStore, TFlashStore and TPackageStore. TMuxStores are used as an interface for TFlashStore and TPackageStore (?) with a monitor. The interface functions are the following (in Protocol order, i.e. in the order you should use to implement another implementation) (this is a dummy implementation sample of the name TSimpleStore) PROTOCOL TSimpleStore : public TStore PROTOCOLVERSION(1.0) { PROTOCOL_IMPL_HEADER_MACRO(TSimpleStore); // static size_t Sizeof(void); // static const TClassInfo* ClassInfo(void); CAPABILITIES( "LOBJ" ) // LOBJ says that this implementation is able to handle large binaries. // If the store is mounted, this is implied somewhere so that it crashes if it wasn't declared. // However, if the store is not registered on the TPSSManager, the large binaries function // will consider this store as non-valid. // (yes, this is my current problem with TATAStore) // Protocols creators and destructors TSimpleStore* New(void); void Delete(void); NewtonErr Init(void* inArg1, unsigned long inArg2, unsigned long inArg3, int inSocketNumber, unsigned long inFlags, void* inArg6); // Initializes the store with the SPSSInfo (inArg6) // Flags are used by TFlashStore to determine if the store is on SRAM, Flash card or Internal Flash NewtonErr NeedsFormat( Boolean* outNeedsFormat ); // Tests if the store needs to be formatted. If so, a dialog will be shown to offer to format it. NewtonErr Format( void ); // Formats the card. Should create an empty object of the ID rootID. NewtonErr GetRootId( unsigned long* ); // There is a root object from which every other objects are linked. NewtonErr NewObject( long inSize, unsigned long* outObjectID ); // Creates a new object. TFlashStore interface calls NewObject( nil, inSIze, outObjectID ); NewtonErr EraseObject( unsigned long inObjectID ); // TFlashStore returns noErr without doing anything. NewtonErr DeleteObject( unsigned long inObjectID ); // Removes the object NewtonErr SetObjectSize( unsigned long inObjectID, long inNewSize ); // Changes the size of an object. Returns kSError_ObjectNotFound if the object cannot be found. NewtonErr GetObjectSize( unsigned long inObjectID, long* outSize ); // Returns the size of an object. Returns kSError_ObjectNotFound if the object cannot be found. NewtonErr Write( unsigned long inObjectID, long inStartOffset, char* inBuffer, long inLength ); // Writes the data for an object from inStartOffset for inLength bytes. NewtonErr Read( unsigned long inObjectID, long inStartOffset, char* outBuffer, long inLength ); // Reads the data for an object from inStartOffset for inLength bytes. NewtonErr GetStoreSizes(long* outTotalSize, long* outUsedSize); // Returns the total size and the used size for the store. NewtonErr IsReadOnly(Boolean*); // Is the store physically read only? NewtonErr LockStore(void); NewtonErr UnlockStore(void); // There is normally a lock store counter. This counter is increased with LockStore and decreased with UnlockStore. NewtonErr Abort(void); // Aborts the current transaction. NewtonErr Idle(unsigned char*, unsigned char*); // TFlashStore does nothing. NewtonErr NextObject(unsigned long inObjectID, unsigned long* outObjectID); // TPackageStore sets outObjectID to 0. Both TFlashStore (which does not set *outObject) and TPackageStore return noErr. NewtonErr CheckIntegrity(unsigned long*); // TFlashStore does nothing. NewtonErr SetBuddy(TStore*); // TMuxStore calls sub store. TFlashStore and TPackageStore return noErr. Boolean OwnsObject(unsigned long inObjectID); // TFlashStore & TPackageStore return true. NewtonErr Address(unsigned long); // TFlashStore and TPackageStore do nothing and return noErr. char* StoreKind(void); // Returns a string describing the store, like "Internal" or "Package" NewtonErr SetStore(TStore* inStore, TObjectID inEnvironment); // Used only for TMuxStore. TFlashStore and TPackageStore returns noErr. Boolean IsSameStore(void*, unsigned long); // TFlashStore does some stuff. Boolean IsLocked(void); // Returns (mLockedStore) NewtonErr VppOff(void); // Any kind of power off, not only Vpp. NewtonErr Sleep(void); // TFlashStore returns noErr Boolean IsROM(void); // A ROM Flash Card is an application card. Package Stores always say they are ROM. // Transaction functions NewtonErr NewWithinTransaction( long inSize, unsigned long* outObjectID ); NewtonErr StartTransactionAgainst( unsigned long inObjectID ); NewtonErr SeparatelyAbort( unsigned long inObjectID ); NewtonErr AddToCurrentTransaction( unsigned long inObjectID ); NewtonErr InSeparateTransaction( unsigned long inObjectID ); NewtonErr LockReadOnly(void); // You should have a counter for locks and unlocks if the store is not hardware locked. Increase it. NewtonErr UnlockReadOnly(Boolean inReset); // If inReset, sets the counter to 0. Otherwise, decreases it if not null. Boolean InTransaction( void ); // Returns whether there is an active transaction. NewtonErr NewObject( char* inBuffer, long inSize, unsigned long* outObjectID ); // Creates a new object with this data. NewtonErr ReplaceObject( unsigned long inObjectID, char* inBuffer, long inSize ); // Replaces an object with this data. NewtonErr CalcXIPObjectSize(long, long, long*); // TPackageStore returns kError_XIP_Not_Possible NewtonErr NewXIPObject(long, unsigned long*); // TPackageStore returns kError_XIP_Not_Possible NewtonErr GetXIPObjectInfo(unsigned long, unsigned long*, unsigned long*, unsigned long*); // TPackageStore returns kError_XIP_Not_Possible } I believe that each store is made of objects. TFlashBlocks (Blocks as referred by Philz) are used because the Flash technology only supports 64 KB erase. This is not the case for Package Stores (read only) and other media (such as ATA, Virtual Stores, etc.). If you want to implement your own TStore, you'll need two vectors that are not in the Public Jump Table. They depend on the machine: (this is only for 2.x machines) TStore::New(char *) MP120 US MP120 D MP130 US MP130 D MP2x00 US MP2100 D eMate 300 01A6A520 ? ? ? 01AFFFF8 01AFEF2C 01AFDEBC TStore::SetStore(TStore *, unsigned long) MP120 US MP120 D MP130 US MP130 D MP2x00 US MP2100 D eMate 300 01A718D4 ? ? ? 01B06348 01B0427C 01B0420C Functionning ============ Objects are simply loaded in memory. Package objects are mapped in virtual memory at 60000000. They are on pages that are loaded when required by the system. There is a huge complex system on top of this interface. There is an ephemrals tracker, objects pipes and large objects handlers. Then only comes the soup and stores. The name of the store is referenced from the root object, with other data such as the store password (has anyone got a password on a store?) TFlashStore manipulates blocks, object references and a cache. NewtonScript interface ====================== You can interface these low-level (and some higher level) APIs from NewtonScript. aStore:ReadObject( objectID, inLength, inOffset ) Reads object ID and returns a binary (of class NIL) with the data. May throw two errors (at least): kSError_ObjectOverRun kSError_ObjectNotFound aStore:GetObjectSize( objectID ) Gets the size of an object. aStore:SetObjectSize( objectID, objectLen ) Credits ======= Philz information confirmed some suppositions I made and completed what I have found. Special thanks to Landon Dyer who saved many hours of work by his precious advises. Special thanks to Konstantinos Xonis who helped me to dig the pointers for the MP2100-D and the eMate 300. ## =========================================================================== ## ## === ALL USERS PLEASE NOTE ======================== ## ## ## ## A new system, the CIRCULATORY system, has been added. ## ## ## ## The long-experimental CIRCULATORY system has been released to users. The ## ## Lisp Machine uses Type B fluid, the L machine uses Type A fluid. When the ## ## switch to Common Lisp occurs both machines will, of course, be Type O. ## ## Please check fluid level by using the DIP stick which is located in the ## ## back of VMI monitors. Unchecked low fluid levels can cause poor paging ## ## performance. ## ## =========================================================================== ##