[android-developers] Re: Initial database population from large data files, several problems
Justin, I get the same exact error as you do. My thinking is it is because our DB files are too big. Have you made any progress on this? Thanks, --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Android Developers" group. To post to this group, send email to android-developers@googlegroups.com To unsubscribe from this group, send email to android-developers-unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/android-developers?hl=en -~--~~~~--~~--~--~---
[android-developers] Re: Initial database population from large data files, several problems
Alistair,
I'm having a little trouble actually implementing the code from that
forum. It's an IOException I'm getting, and I've tracked it down to the
exact line in my code.
What I've done so far:
1. created the database file named wordlist.db and placed it in the
assets folder of my Eclipse Package Explorer Tree. (I'm certain this
database file is good--at least on my desktop machine--who knows what's
happening when it goes to the phone)
2. adapted the code to my purose.
3. after getting an IOException, I commented out all the lines of code
in the method copyDataBase, and then added them back in one at a time
until the exception was thrown again.
4. I simplified the code just to be sure and removed the while loop,
replacing it with: myInput.read(buffer); but I still throw an error when
this line is not commented out.
Is there a way to see a more specific error than just IOException, and
if not, what might be a possible cause for being unable to read a byte
block from the file?
My code:
public void onCreate(SQLiteDatabase db) {
try {
copyDataBase();
} catch (IOException e) {
throw new Error("Error copying database " + e);
}
}
private void copyDataBase() throws IOException{
//Open your local db as the input stream
InputStream myInput = myContext.getAssets().open("wordlist.db");
// Path to the just created empty db
String outFileName = DATABASE_PATH + "wordlist.db";
//Open the empty db as the output stream
//OutputStream myOutput = new FileOutputStream(outFileName);
//transfer bytes from the inputfile to the outputfile
byte[] buffer = new byte[1024];
int length;
while ((length = myInput.read(buffer)) > 0) {
//myOutput.write(buffer, 0, length);
}
//Close the streams
//myOutput.flush();
//myOutput.close();
myInput.close();
}
Alistair. wrote:
> Justin,
>
> You might find this article of interest. This is a technique to pre-
> populate the database and bundle into the assets directory of your
> application.
>
> http://www.reigndesign.com/blog/using-your-own-sqlite-database-in-android-applications/
>
> Al.
>
> On Mar 5, 7:49 am, Justin Allen Jaynes wrote:
>
>> I'm building a dictionary application with 135,000 word entries (words
>> only). My raw file must have been too large (1.5 meg), because I got
>> this error:
>>
>> D/asset (909): Data exceeds UNCOMPRESS_DATA_MAX (1424000 vs 1048576)
>>
>> I've searched for this error with very few relevant hits. It seemed to
>> mean I could not open an uncompressed file over a meg. So I then split
>> the file into two smaller files and ran my code on both of them. It
>> worked out fine. My total application size is 3 meg installed.
>>
>> My code is:
>> public void onCreate(SQLiteDatabase database) {
>> database.execSQL("CREATE TABLE " + DATABASE_TABLE + " (wordid
>> INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, word VARCHAR);");
>>
>> Scanner fileScanner = new
>> Scanner(myContext.getResources().openRawResource(R.raw.wordlist));
>> while ( fileScanner.hasNextLine() ) {
>> String wordFromFile = fileScanner.nextLine();
>> database.execSQL("INSERT INTO words (word) VALUES ('" +
>> wordFromFile + "');");
>> }
>> fileScanner = new
>> Scanner(myContext.getResources().openRawResource(R.raw.wordlist2));
>> while ( fileScanner.hasNextLine() ) {
>> String wordFromFile = fileScanner.nextLine();
>> database.execSQL("INSERT INTO words (word) VALUES ('" +
>> wordFromFile + "');");
>> }
>>
>> }
>>
>> However, when the application is first run, it takes several MINUTES to
>> initialize the database in this way. Is there a way (like a copy
>> command, as found in, say, postgresql, or a restore of a database file)
>> to copy data from a raw file, and can such a method be accessed from the
>> SDK so that standard first-run procedures can correctly set up the
>> database? I have been unable to locate such a luxury. I am seeking to
>> speed up this data populating process.
>>
>> First question: how can I speed up my database population?
>>
>> Second question: is there a way to read a raw resource file larger than
>> 1 megabyte (aside from making it into two smaller files)? If not, why?
>>
>> Justin
>>
> >
>
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google
Groups "Android Developers" group.
To post to this group, send email to android-developers@googlegroups.com
To unsubscribe from this group, send email to
android-developers-unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/android-developers?hl=en
-~--~~~~--~~--~--~---
[android-developers] Re: Initial database population from large data files, several problems
Alistair,
This is precisely the approach I was looking for. Thank you--most
invaluable to me.
Justin
Alistair. wrote:
> Justin,
>
> You might find this article of interest. This is a technique to pre-
> populate the database and bundle into the assets directory of your
> application.
>
> http://www.reigndesign.com/blog/using-your-own-sqlite-database-in-android-applications/
>
> Al.
>
> On Mar 5, 7:49 am, Justin Allen Jaynes wrote:
>
>> I'm building a dictionary application with 135,000 word entries (words
>> only). My raw file must have been too large (1.5 meg), because I got
>> this error:
>>
>> D/asset (909): Data exceeds UNCOMPRESS_DATA_MAX (1424000 vs 1048576)
>>
>> I've searched for this error with very few relevant hits. It seemed to
>> mean I could not open an uncompressed file over a meg. So I then split
>> the file into two smaller files and ran my code on both of them. It
>> worked out fine. My total application size is 3 meg installed.
>>
>> My code is:
>> public void onCreate(SQLiteDatabase database) {
>> database.execSQL("CREATE TABLE " + DATABASE_TABLE + " (wordid
>> INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, word VARCHAR);");
>>
>> Scanner fileScanner = new
>> Scanner(myContext.getResources().openRawResource(R.raw.wordlist));
>> while ( fileScanner.hasNextLine() ) {
>> String wordFromFile = fileScanner.nextLine();
>> database.execSQL("INSERT INTO words (word) VALUES ('" +
>> wordFromFile + "');");
>> }
>> fileScanner = new
>> Scanner(myContext.getResources().openRawResource(R.raw.wordlist2));
>> while ( fileScanner.hasNextLine() ) {
>> String wordFromFile = fileScanner.nextLine();
>> database.execSQL("INSERT INTO words (word) VALUES ('" +
>> wordFromFile + "');");
>> }
>>
>> }
>>
>> However, when the application is first run, it takes several MINUTES to
>> initialize the database in this way. Is there a way (like a copy
>> command, as found in, say, postgresql, or a restore of a database file)
>> to copy data from a raw file, and can such a method be accessed from the
>> SDK so that standard first-run procedures can correctly set up the
>> database? I have been unable to locate such a luxury. I am seeking to
>> speed up this data populating process.
>>
>> First question: how can I speed up my database population?
>>
>> Second question: is there a way to read a raw resource file larger than
>> 1 megabyte (aside from making it into two smaller files)? If not, why?
>>
>> Justin
>>
> >
>
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google
Groups "Android Developers" group.
To post to this group, send email to android-developers@googlegroups.com
To unsubscribe from this group, send email to
android-developers-unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/android-developers?hl=en
-~--~~~~--~~--~--~---
[android-developers] Re: Initial database population from large data files, several problems
On Mar 4, 11:49 pm, Justin Allen Jaynes wrote:
> I'm building a dictionary application with 135,000 word entries (words
> only). My raw file must have been too large (1.5 meg), because I got
> this error:
>
> D/asset (909): Data exceeds UNCOMPRESS_DATA_MAX (1424000 vs 1048576)
>
> I've searched for this error with very few relevant hits. It seemed to
> mean I could not open an uncompressed file over a meg. So I then split
> the file into two smaller files and ran my code on both of them. It
> worked out fine. My total application size is 3 meg installed.
>
> My code is:
> public void onCreate(SQLiteDatabase database) {
> database.execSQL("CREATE TABLE " + DATABASE_TABLE + " (wordid
> INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, word VARCHAR);");
>
> Scanner fileScanner = new
> Scanner(myContext.getResources().openRawResource(R.raw.wordlist));
> while ( fileScanner.hasNextLine() ) {
> String wordFromFile = fileScanner.nextLine();
> database.execSQL("INSERT INTO words (word) VALUES ('" +
> wordFromFile + "');");
> }
> fileScanner = new
> Scanner(myContext.getResources().openRawResource(R.raw.wordlist2));
> while ( fileScanner.hasNextLine() ) {
> String wordFromFile = fileScanner.nextLine();
> database.execSQL("INSERT INTO words (word) VALUES ('" +
> wordFromFile + "');");
> }
>
> }
>
> However, when the application is first run, it takes several MINUTES to
> initialize the database in this way. Is there a way (like a copy
> command, as found in, say, postgresql, or a restore of a database file)
> to copy data from a raw file, and can such a method be accessed from the
> SDK so that standard first-run procedures can correctly set up the
> database? I have been unable to locate such a luxury. I am seeking to
> speed up this data populating process.
>
> First question: how can I speed up my database population?
Pre-populating it and building the database file into your app, as the
other response suggested, is probably the best way to do it.
However, if you decide to populate the database when your app is first
run, you might still be able to speed it up by wrapping the whole
process inside an SQLite transaction. Otherwise, it creates a separate
transaction for each query, which is slow. That's according to the
SQLite Optimization FAQ (http://web.utk.edu/~jplyon/sqlite/
SQLite_optimization_FAQ.html), although that FAQ is out of date by now
so maybe it's no longer true.
Jesse
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google
Groups "Android Developers" group.
To post to this group, send email to android-developers@googlegroups.com
To unsubscribe from this group, send email to
android-developers-unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/android-developers?hl=en
-~--~~~~--~~--~--~---
[android-developers] Re: Initial database population from large data files, several problems
Justin,
You might find this article of interest. This is a technique to pre-
populate the database and bundle into the assets directory of your
application.
http://www.reigndesign.com/blog/using-your-own-sqlite-database-in-android-applications/
Al.
On Mar 5, 7:49 am, Justin Allen Jaynes wrote:
> I'm building a dictionary application with 135,000 word entries (words
> only). My raw file must have been too large (1.5 meg), because I got
> this error:
>
> D/asset (909): Data exceeds UNCOMPRESS_DATA_MAX (1424000 vs 1048576)
>
> I've searched for this error with very few relevant hits. It seemed to
> mean I could not open an uncompressed file over a meg. So I then split
> the file into two smaller files and ran my code on both of them. It
> worked out fine. My total application size is 3 meg installed.
>
> My code is:
> public void onCreate(SQLiteDatabase database) {
> database.execSQL("CREATE TABLE " + DATABASE_TABLE + " (wordid
> INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, word VARCHAR);");
>
> Scanner fileScanner = new
> Scanner(myContext.getResources().openRawResource(R.raw.wordlist));
> while ( fileScanner.hasNextLine() ) {
> String wordFromFile = fileScanner.nextLine();
> database.execSQL("INSERT INTO words (word) VALUES ('" +
> wordFromFile + "');");
> }
> fileScanner = new
> Scanner(myContext.getResources().openRawResource(R.raw.wordlist2));
> while ( fileScanner.hasNextLine() ) {
> String wordFromFile = fileScanner.nextLine();
> database.execSQL("INSERT INTO words (word) VALUES ('" +
> wordFromFile + "');");
> }
>
> }
>
> However, when the application is first run, it takes several MINUTES to
> initialize the database in this way. Is there a way (like a copy
> command, as found in, say, postgresql, or a restore of a database file)
> to copy data from a raw file, and can such a method be accessed from the
> SDK so that standard first-run procedures can correctly set up the
> database? I have been unable to locate such a luxury. I am seeking to
> speed up this data populating process.
>
> First question: how can I speed up my database population?
>
> Second question: is there a way to read a raw resource file larger than
> 1 megabyte (aside from making it into two smaller files)? If not, why?
>
> Justin
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google
Groups "Android Developers" group.
To post to this group, send email to android-developers@googlegroups.com
To unsubscribe from this group, send email to
android-developers-unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/android-developers?hl=en
-~--~~~~--~~--~--~---
