Re: [Firebird-devel] Firebird 2.1 External Table Files Internal Format
Any suggestions, guys? Can anyone please point us to the right place in the source code so we can have a closer look? Also, it appears that external table doesn't like a row being over 32k (including padding) in the external file. When we have rows over 32k, select from the external table just returns nothing. Is this a known limitation? I couldn't find this mentioned in any documentation. We are running Firebird 2.1.2 64bit Classic Server on Centos 6. On 14 September 2013 19:24, Huan Ruan leohuanr...@gmail.com wrote: Hi All We are trying to implement a bulk data importer by writing our own external table files. We've tried storing all fields as text, but due to the volume of data, this put us well over the 2GB external file size limit in Firebird 2.1. So we've resorted to saving the files in Firebird's own binary format. To simplify things a little, we only need to deal with CHAR, NUMERIC and TIMESTAMP fields. The data formats were easy enough to figure out. However, the byte alignment rules for some field types are a little baffling. From what I've seen: - CHAR fields can start at any position within a row - 2-byte NUMERIC fields can start at bytes 0, 2, 4... - 4-byte NUMERIC fields can start at bytes 0, 4, 8... - 8-byte NUMERIC / TIMESTAMP fields: -- Can start at bytes 0, 8, 16... as long as either: --- The first field in the table is an 8-byte NUMERIC / TIMESTAMP, or --- The table contains more than 32 fields in total -- Otherwise, can start at bytes 4, 12, 20... Is this right? Is there anything I'm missing? We didn't get very far digging around in the source code. Thanks Huan After a bit more testing, it looks like there's a bit more to the 8-byte NUMERIC / TIMESTAMP rule. The fields will start at bytes 0, 8, 16... if the total number of fields is between 33 and 64, between 97 and 128, etc. In other words, if floor((FieldCount - 1)/32) is odd. -- October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register http://pubads.g.doubleclick.net/gampad/clk?id=60133471iu=/4140/ostg.clktrkFirebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] Firebird 2.1 External Table Files InternalFormat
Any suggestions, guys? Can anyone please point us to the right place in the source code so we can have a closer look? Physical layout of unpacked record in memory is simple: - bitmak of NULL values, one bit per field This bitmask contains one 32-bit intereg per every 32 fields in relation. - data, field by field, aligned by value evaluated from data type and declared data length. This alignment rules could be found at MET_align() function. It returns 1 for CHAR type, 2 for VARCHAR type and MIN(data_length, 8) for all other data types. Firebird used almost the same format for external table's record as for its internal records with only distinctions: external record have no NULL's bitmask. But offsets of fields in record are evaluated by the common rules. Its just adjusted when external table is accessed. For example CREATE TABLE t1 ( ID BIGINT, NAME VARCHAR(32), FOO INT ) The record layout is offsetlengthfield_name 0 4nulls mask 8 8ID 16 34NAME 52 4FOO Record length is 56 bytes. Notes: - offset of ID is 8 and not 4 as 8-bytes field must be aligned at 8-byte position, - length of VARCHAR field included additional 2 bytes for real data length (32+2 = 34) - offset of FOO is 52 and not 50 (34+16) as it must be aligned at 4-bytes position Above is so called format of record and it is used everywhere across the engine. Engine calculates and store this format in RDB$FORMATS and used it when table is accesses. But EXTERNAL tables have no NULL's bitmask therefore when data is mapped from\to record buffer engine adjusted fields offsets in format by substracting offset of a 1st field in format (not size of NULL's mask!). So, in fact offsets used to access external table with structure above (i.e. on-disk record representation) is: offsetlengthfield_name 0 8ID 8 34NAME 46 4FOO and it could look strange from alignment point of view. Also, it appears that external table doesn't like a row being over 32k (including padding) in the external file. When we have rows over 32k, select from the external table just returns nothing. Is this a known limitation? I couldn't find this mentioned in any documentation. We are running Firebird 2.1.2 64bit Classic Server on Centos 6. I'm not sure FB 2.1.x could correctly work with 32K+ records. But i can be wrong, i don't remember exactly. Could you check it with FB 2.5 ? Hope it helps, Vlad -- October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register http://pubads.g.doubleclick.net/gampad/clk?id=60133471iu=/4140/ostg.clktrk Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] building fbclient for android arm
On 09/25/13 19:42, marius adrian popa wrote: I have an error error: 'PTHREAD_PRIO_INHERIT' was not declared in this scope /home/mariuz/work/firebird-trunk/src/common/isc_sync.cpp: In constructor 'Firebird::SharedMemoryBase::SharedMemoryBase(const TEXT*, ULONG, Firebird::IpcObject*)': /home/mariuz/work/firebird-trunk/src/common/isc_sync.cpp:1989:61: error: 'PTHREAD_PRIO_INHERIT' was not declared in this scope /home/mariuz/work/firebird-trunk/src/common/isc_sync.cpp:1989:81: error: 'pthread_mutexattr_setprotocol' was not declared in this scope Try with current trunk. But must say that isc_sync.cpp requires serious rework to make engine run on Android. -- October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register http://pubads.g.doubleclick.net/gampad/clk?id=60133471iu=/4140/ostg.clktrk Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
Re: [Firebird-devel] building fbclient for android arm
Thanks all worked with export NDK=/home/mariuz/android-ndk-r9 ./configure --with-system-editline --enable-binreloc --with-cross-build=android.arme On Thu, Sep 26, 2013 at 11:07 AM, Alex Peshkoff peshk...@mail.ru wrote: On 09/25/13 19:42, marius adrian popa wrote: I have an error error: 'PTHREAD_PRIO_INHERIT' was not declared in this scope /home/mariuz/work/firebird-trunk/src/common/isc_sync.cpp: In constructor 'Firebird::SharedMemoryBase::SharedMemoryBase(const TEXT*, ULONG, Firebird::IpcObject*)': /home/mariuz/work/firebird-trunk/src/common/isc_sync.cpp:1989:61: error: 'PTHREAD_PRIO_INHERIT' was not declared in this scope /home/mariuz/work/firebird-trunk/src/common/isc_sync.cpp:1989:81: error: 'pthread_mutexattr_setprotocol' was not declared in this scope Try with current trunk. But must say that isc_sync.cpp requires serious rework to make engine run on Android. -- October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register http://pubads.g.doubleclick.net/gampad/clk?id=60133471iu=/4140/ostg.clktrk Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel -- October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register http://pubads.g.doubleclick.net/gampad/clk?id=60133471iu=/4140/ostg.clktrk Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
