Re: [Virtuoso-users] virtuoso striping
Yes, you can "split" a large file into many small files. At the colo, you can put them back together again. The command to put them back together is "cat". The "join" command does something else, so you don't want to try to use it. NB: this is actually what the cat command is for. "cat" is short for "concatenate". Although it's rarely used for this purpose! ;) Alternatively, there are options for rsync that turn off the checksum stuff. So if a file transfer gets interrupted, it picks off right where it left off. You can then do file verification outside the scope of rsync, e.g. by doing sha1sum on both sides and comparing the results. Contact your local sysadmins for assistance with either of these options. :) - Morty On Wed, Apr 15, 2015 at 02:50:31PM -0400, Gang Fu wrote: > We want to transfer the files to another location, 'colo' for disaster > recovery. The long distance transfer is time-consuming and may fail > sometimes. > > We are using rsync, and we believe rsync a 500 GB file or rsync many small > files indeed make difference, since rsync does a checksum validation before > transfer, so if a large portion of many small files have the same checksum, > then we only need to transfer a small port of them. > > Can we just 'split' and 'join' db files before and after transferring? > > Best, > Gang > > On Wed, Apr 15, 2015 at 1:17 PM, Morty wrote: > > > On Tue, Apr 14, 2015 at 12:24:22PM -0400, Gang Fu wrote: > > > > > We want to copy a large virtuoso db from one server to another in > > > different location. We cannot copy single 500 GB db file, which is > > > slow and unstable. So we want to break the db files in different > > > segments. I have tried with virtuoso striping: each segment has 20 > > > GB, and in total we have over 25 segments. > > > > What issue are you seeing with transferring a 500GB file? > > Transferring one 500GB file should not be significantly slower than > > transferring 25x 20GB files. > > > > If you are concerned about a transfer interruption, you could use > > rsync. rsync has options to resume a failed transfer. > > > > Alternatively, you could use the Linux/Unix "split" command to split > > the one large file into a bunch of smaller files. > > > > Or you could use the commercial version of virtuoso with built-in > > replication. > > > > - Morty > > -- Mordechai T. Abzug Linux red-sonja 3.11.0-24-generic #42-Ubuntu SMP Fri Jul 4 21:19:31 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux "A verbal contract isn't worth the paper it's written on." - Samuel Goldwyn -- BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT Develop your own process in accordance with the BPMN 2 standard Learn Process modeling best practices with Bonita BPM through live exercises http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_ source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF ___ Virtuoso-users mailing list Virtuoso-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/virtuoso-users
Re: [Virtuoso-users] virtuoso striping
On Tue, Apr 14, 2015 at 12:24:22PM -0400, Gang Fu wrote: > We want to copy a large virtuoso db from one server to another in > different location. We cannot copy single 500 GB db file, which is > slow and unstable. So we want to break the db files in different > segments. I have tried with virtuoso striping: each segment has 20 > GB, and in total we have over 25 segments. What issue are you seeing with transferring a 500GB file? Transferring one 500GB file should not be significantly slower than transferring 25x 20GB files. If you are concerned about a transfer interruption, you could use rsync. rsync has options to resume a failed transfer. Alternatively, you could use the Linux/Unix "split" command to split the one large file into a bunch of smaller files. Or you could use the commercial version of virtuoso with built-in replication. - Morty -- BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT Develop your own process in accordance with the BPMN 2 standard Learn Process modeling best practices with Bonita BPM through live exercises http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_ source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF ___ Virtuoso-users mailing list Virtuoso-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/virtuoso-users
Re: [Virtuoso-users] create procedure in bash shell
On Tue, Apr 07, 2015 at 04:36:39PM -0400, Gang Fu wrote: > In the bash shell, we need to take care of single quote by converting ' to > '"'"'. Gang -- You can avoid some issues and debug more easily by using isql's run file syntax. temp_file=`mktemp /somedir/tmpX` $command_to_generate_procedure > $temp_file && isql dba $dba_password $temp_file If there are any issues, you can look at the temporary file to debug quoting and the like. - Morty -- BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT Develop your own process in accordance with the BPMN 2 standard Learn Process modeling best practices with Bonita BPM through live exercises http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_ source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF ___ Virtuoso-users mailing list Virtuoso-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/virtuoso-users
Re: [Virtuoso-users] virtuoso isql batch mode?
Hugh -- To clarify, I mean that I just want the SQL output without any headers or footers. This is for sysadmin-type scripting rather than for actual DB use. For example, the command: echo "select U_NAME FROM SYS_USERS WHERE U_ID=106;"|\ /opt/virtuoso/bin/isql dba $dba_password results in: Connected to OpenLink Virtuoso Driver: 07.20.3212 OpenLink Virtuoso ODBC Driver OpenLink Interactive SQL (Virtuoso), version 0.9849b. Type HELP; for help and EXIT; to exit. SQL> U_NAME VARCHAR NOT NULL ___ SPARQL 1 Rows. -- 1 msec. SQL> What I want is to get just the actual one line of output: SPARQL I can improve on the above somewhat through an approach like this: $ cat foo.sql set MESSAGE OFF; set VERBOSE OFF; set BANNER OFF; select U_NAME FROM SYS_USERS WHERE U_ID=106; $ /opt/virtuoso/bin/isql dba `cat $dba_password_file` foo.sql Connected to OpenLink Virtuoso Driver: 07.20.3212 OpenLink Virtuoso ODBC Driver SPARQL But I still get the "Connected" and "Driver:" output. The isql variables / macros are set too late to help me. So presumably, a command-line option is needed. It is possible to script around this with suitable head, tail, and grep pipe postprocessing. However, those kinds of scripts tend to be fragile in the face of software upgrades. The formats tend to change subtly between versions. So DB systems often provide a command-line option that just outputs raw data. sybase's isql command has such an option. mysql's mysql command has such an option. So does postgres's command. It would be nice if virtuoso's isql did as well, if it doesn't already. Thanks! - Morty On Tue, Apr 07, 2015 at 12:39:32AM +0100, Hugh Williams wrote: > Hi Morty, > > Not sure what you mean by "Batch mode”, Virtuoso has a “load ;” > command that can be used for executing multiple isql commands at once or are > you seeking to be able to load a series sql command as part of one > transaction and commit when done ? > > Best Regards > Hugh Williams > Professional Services > OpenLink Software, Inc. // http://www.openlinksw.com/ > Weblog -- http://www.openlinksw.com/blogs/ > LinkedIn -- http://www.linkedin.com/company/openlink-software/ > Twitter -- http://twitter.com/OpenLink > Google+ -- http://plus.google.com/100570109519069333827/ > Facebook -- http://www.facebook.com/OpenLinkSoftware > Universal Data Access, Integration, and Management Technology Providers > > > On 4 Apr 2015, at 20:59, Morty wrote: > > > > Is there some way to put virtuoso isql into a true batch mode? Some > > other database system command-line interfaces provide a batch mode > > option that just displays results, not version banners, table headers, > > or time information. This is very handy for building scripts around. > > > > I can sort-of emulate this in isql by setting: > > > > set VERBOSE OFF; > > set BANNER OFF; > > set MESSAGE OFF; > > > > But I still get the "Connected to" and "Driver" lines. And I'm also > > concerned that later versions could add new sources of information. > > So it would be nice to have a command-line option that will set the > > options necessary to script cleanly, both now and in the future. > > > > Hopefully such a thing already exists, and I just didn't find it. If > > not, feature request: please implement such a mechanism. > > > > Note: this is currently under virtuoso open source version 7.2.0, > > under Linux. The commercial version is not an option at this time. > > > > - Morty > > > > -- > > Dive into the World of Parallel Programming The Go Parallel Website, > > sponsored > > by Intel and developed in partnership with Slashdot Media, is your hub for > > all > > things parallel software development, from weekly thought leadership blogs > > to > > news, videos, case studies, tutorials and more. Take a look and join the > > conversation now. http://goparallel.sourceforge.net/ > > ___ > > Virtuoso-users mailing list > > Virtuoso-users@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/virtuoso-users > -- Mordechai T. Abzug Linux red-sonja 3.11.0-24-generic #42-Ubuntu SMP Fri Jul 4 21:19:31 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux "A billion here, a billion there, sooner or later it adds up to real money." -Everett Dirksen -- BPM Camp - Fre
[Virtuoso-users] virtuoso isql batch mode?
Is there some way to put virtuoso isql into a true batch mode? Some other database system command-line interfaces provide a batch mode option that just displays results, not version banners, table headers, or time information. This is very handy for building scripts around. I can sort-of emulate this in isql by setting: set VERBOSE OFF; set BANNER OFF; set MESSAGE OFF; But I still get the "Connected to" and "Driver" lines. And I'm also concerned that later versions could add new sources of information. So it would be nice to have a command-line option that will set the options necessary to script cleanly, both now and in the future. Hopefully such a thing already exists, and I just didn't find it. If not, feature request: please implement such a mechanism. Note: this is currently under virtuoso open source version 7.2.0, under Linux. The commercial version is not an option at this time. - Morty -- Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ ___ Virtuoso-users mailing list Virtuoso-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/virtuoso-users
Re: [Virtuoso-users] virtuoso isql password file option
On Fri, Mar 20, 2015 at 07:26:07AM -0400, Kingsley Idehen wrote: > Some of us see the perceived "simplicity" of passwords as the mother > of all vulnerabilities, in a network environment. We're not using isql over the network, only locally. isql is only used to drive the scripts that configure virtuoso and load data. The scripts themselves are delivered via ssh (typically pubkey auth) and/or puppet over https (with client certs). Actual network data use is over a SPARQL endpoint. Which is read-only. For our usage model, a password-based approach doesn't seem (too) bad. certs are indeed easy to generate. But they require ongoing monitoring and management to deal with cert expiration. I'm happy to deal with that when the security model warrants it, but prefer to avoid it otherwise. > There is an isql-odbc variant of isql which can read passwords from > an INI file. Currently, that's missing from our releases, but it > will be added, following this conversation. Excellent! Thanks. :) - Morty -- Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ ___ Virtuoso-users mailing list Virtuoso-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/virtuoso-users
Re: [Virtuoso-users] virtuoso isql password file option
On Thu, Mar 19, 2015 at 07:13:43AM -0400, Kingsley Idehen wrote: > You appear to assume that iSQL can only be used to connect to a > Virtuoso instance using a single approach? Here an option dump > (note: we even support use of pkcs#12 and pem files for secure > connections that don't used passwords at all: I'm aware that other authentication options are available. However, passwords are very simple to manage. Given that passwords are present and supported, my request is to implement them in a (slightly more) secure way. Why implement passwords and then hobble them? Other database systems I've worked with have had mechanisms to deal with this problem. Usually by having a "password file" option or a configuration file option that could hold a password. - Morty -- Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ ___ Virtuoso-users mailing list Virtuoso-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/virtuoso-users
[Virtuoso-users] virtuoso isql password file option
virtuoso isql's use of a command-line argument to specify the password is problematic for security. This means that user passwords are visible via "ps". It also means that orchestration systems that log command-lines have passwords in the logs. I'd really like to get away from this. I can (somewhat) hide the password by putting it in a file and utilizing `cat $dba_password_file` in scripts, but it's still visible via ps, and it's clunky. Is there any baked-in way to specify the password via a password file, environment variable, or other mechanism? Hopefully such a thing already exists, and I just didn't find it. If not, feature request: please implement such a mechanism. [Or would you guys accept a patch to do this? I'm not much of a C coder, but I think I could handle something this trivial.] Note: this is currently under virtuoso open source version 7.1.0, under Linux. I'm sure we'll be upgrading to the most recent version shortly. But I don't see any changes to isql under the 7.2.0 release notes, so I'm assuming "upgrade to 7.2.0" won't solve my problems. The commercial version is not an option at this time. - Morty -- Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ ___ Virtuoso-users mailing list Virtuoso-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/virtuoso-users
