I meant to reply earlier that the current DataStax doc on EC2 is actually reasonably decent. It says this about EBS:
"SSD-backed general purpose volumes (GP2) or provisioned IOPS volumes (PIOPS) are suitable for production workloads." with the caveat of: "EBS magnetic volumes are not recommended for Cassandra data storage volumes for the following reasons:..." as well as: "Note: Use only ephemeral instance-store or the recommended EBS volume types for Cassandra data storage." See: http://docs.datastax.com/en/cassandra/3.x/cassandra/planning/planPlanningEC2.html -- Jack Krupansky On Wed, Feb 3, 2016 at 7:27 PM, Sebastian Estevez < sebastian.este...@datastax.com> wrote: > Good points Bryan, some more color: > > Regular EBS is *not* okay for C*. But AWS has some nicer EBS now that has > performed okay recently: > > http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html > > https://www.youtube.com/watch?v=1R-mgOcOSd4 > > > The cloud vendors are moving toward shared storage and we can't ignore > that in the long term (they will push us in that direction financially). > Fortunately their shared storage offerings are also getting better. For > example google's elastic storage offerring provides very reliable > latencies <https://www.youtube.com/watch?v=qf-7IhCqCcI>which is what we > care the most about, not iops. > > On the practical side, a key thing I've noticed with real deployments is > that the size of the volume affects how fast it will perform and how stable > it's latencies will be so make sure to get large EBS volumes > 1tb to get > decent performance, even if your nodes aren't that dense. > > > > > All the best, > > > [image: datastax_logo.png] <http://www.datastax.com/> > > Sebastián Estévez > > Solutions Architect | 954 905 8615 | sebastian.este...@datastax.com > > [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image: > facebook.png] <https://www.facebook.com/datastax> [image: twitter.png] > <https://twitter.com/datastax> [image: g+.png] > <https://plus.google.com/+Datastax/about> > <http://feeds.feedburner.com/datastax> > <http://goog_410786983> > > > <http://www.datastax.com/gartner-magic-quadrant-odbms> > > DataStax is the fastest, most scalable distributed database technology, > delivering Apache Cassandra to the world’s most innovative enterprises. > Datastax is built to be agile, always-on, and predictably scalable to any > size. With more than 500 customers in 45 countries, DataStax is the > database technology and transactional backbone of choice for the worlds > most innovative companies such as Netflix, Adobe, Intuit, and eBay. > > On Wed, Feb 3, 2016 at 7:23 PM, Bryan Cheng <br...@blockcypher.com> wrote: > >> From my experience, EBS has transitioned from "stay the hell away" to >> "OK" as the new GP2 SSD type has come out and stabilized over the last few >> years, especially with the addition of EBS-optimized instances that have >> dedicated EBS bandwidth. The latter has really helped to stabilize the >> problematic 99.9-percentile latency spikes that use to plague EBS volumes. >> >> EBS (IMHO) has always had operational advantages, but inconsistent >> latency and generally poor performance in the past lead many to disregard >> it. >> >> On Wed, Feb 3, 2016 at 4:09 PM, James Rothering <jrother...@codojo.me> >> wrote: >> >>> Just curious here ... when did EBS become OK for C*? Didn't they always >>> push towards using ephemeral disks? >>> >>> On Wed, Feb 3, 2016 at 12:17 PM, Ben Bromhead <b...@instaclustr.com> >>> wrote: >>> >>>> For what it's worth we've tried d2 instances and they encourage >>>> terrible things like super dense nodes (increases your replacement time). >>>> In terms of useable storage I would go with gp2 EBS on a m4 based instance. >>>> >>>> On Mon, 1 Feb 2016 at 14:25 Jack Krupansky <jack.krupan...@gmail.com> >>>> wrote: >>>> >>>>> Ah, yes, the good old days of m1.large. >>>>> >>>>> -- Jack Krupansky >>>>> >>>>> On Mon, Feb 1, 2016 at 5:12 PM, Jeff Jirsa <jeff.ji...@crowdstrike.com >>>>> > wrote: >>>>> >>>>>> A lot of people use the old gen instances (m1 in particular) because >>>>>> they came with a ton of effectively free ephemeral storage (up to 1.6TB). >>>>>> Whether or not they’re viable is a decision for each user to make. >>>>>> They’re >>>>>> very, very commonly used for C*, though. At a time when EBS was not >>>>>> sufficiently robust or reliable, a cluster of m1 instances was the de >>>>>> facto >>>>>> standard. >>>>>> >>>>>> The canonical “best practice” in 2015 was i2. We believe we’ve made a >>>>>> compelling argument to use m4 or c4 instead of i2. There exists a company >>>>>> we know currently testing d2 at scale, though I’m not sure they have much >>>>>> in terms of concrete results at this time. >>>>>> >>>>>> - Jeff >>>>>> >>>>>> From: Jack Krupansky >>>>>> Reply-To: "user@cassandra.apache.org" >>>>>> Date: Monday, February 1, 2016 at 1:55 PM >>>>>> >>>>>> To: "user@cassandra.apache.org" >>>>>> Subject: Re: EC2 storage options for C* >>>>>> >>>>>> Thanks. My typo - I referenced "C2 Dense Storage" which is really "D2 >>>>>> Dense Storage". >>>>>> >>>>>> The remaining question is whether any of the "Previous Generation >>>>>> Instances" should be publicly recommended going forward. >>>>>> >>>>>> And whether non-SSD instances should be recommended going forward as >>>>>> well. sure, technically, someone could use the legacy instances, but the >>>>>> question is what we should be recommending as best practice going >>>>>> forward. >>>>>> >>>>>> Yeah, the i2 instances look like the sweet spot for any non-EBS >>>>>> clusters. >>>>>> >>>>>> -- Jack Krupansky >>>>>> >>>>>> On Mon, Feb 1, 2016 at 4:30 PM, Steve Robenalt < >>>>>> sroben...@highwire.org> wrote: >>>>>> >>>>>>> Hi Jack, >>>>>>> >>>>>>> At the bottom of the instance-types page, there is a link to the >>>>>>> previous generations, which includes the older series (m1, m2, etc), >>>>>>> many >>>>>>> of which have HDD options. >>>>>>> >>>>>>> There are also the d2 (Dense Storage) instances in the current >>>>>>> generation that include various combos of local HDDs. >>>>>>> >>>>>>> The i2 series has good sized SSDs available, and has the advanced >>>>>>> networking option, which is also useful for Cassandra. The enhanced >>>>>>> networking is available with other instance types as well, as you'll >>>>>>> see on >>>>>>> the feature list under each type. >>>>>>> >>>>>>> Steve >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, Feb 1, 2016 at 1:17 PM, Jack Krupansky < >>>>>>> jack.krupan...@gmail.com> wrote: >>>>>>> >>>>>>>> Thanks. Reading a little bit on AWS, and back to my SSD vs. >>>>>>>> magnetic question, it seems like magnetic (HDD) is no longer a >>>>>>>> recommended >>>>>>>> storage option for databases on AWS. In particular, only the C2 Dense >>>>>>>> Storage instances have local magnetic storage - all the other instance >>>>>>>> types are SSD or EBS-only - and EBS Magnetic is only recommended for >>>>>>>> "Infrequent Data Access." >>>>>>>> >>>>>>>> For the record, that AWS doc has Cassandra listed as a use case for >>>>>>>> i2 instance types. >>>>>>>> >>>>>>>> Also, the AWS doc lists EBS io2 for the NoSQL database use case and >>>>>>>> gp2 only for the "small to medium databases" use case. >>>>>>>> >>>>>>>> Do older instances with local HDD still exist on AWS (m1, m2, >>>>>>>> etc.)? Is the doc simply for any newly started instances? >>>>>>>> >>>>>>>> See: >>>>>>>> https://aws.amazon.com/ec2/instance-types/ >>>>>>>> http://aws.amazon.com/ebs/details/ >>>>>>>> >>>>>>>> >>>>>>>> -- Jack Krupansky >>>>>>>> >>>>>>>> On Mon, Feb 1, 2016 at 2:09 PM, Jeff Jirsa < >>>>>>>> jeff.ji...@crowdstrike.com> wrote: >>>>>>>> >>>>>>>>> > My apologies if my questions are actually answered on the video >>>>>>>>> or slides, I just did a quick scan of the slide text. >>>>>>>>> >>>>>>>>> Virtually all of them are covered. >>>>>>>>> >>>>>>>>> > I'm curious where the EBS physical devices actually reside - are >>>>>>>>> they in the same rack, the same data center, same availability zone? I >>>>>>>>> mean, people try to minimize network latency between nodes, so how >>>>>>>>> exactly >>>>>>>>> is EBS able to avoid network latency? >>>>>>>>> >>>>>>>>> Not published,and probably not a straight forward answer (probably >>>>>>>>> have redundancy cross-az, if it matches some of their other published >>>>>>>>> behaviors). The promise they give you is ‘iops’, with a certain block >>>>>>>>> size. >>>>>>>>> Some instance types are optimized for dedicated, ebs-only network >>>>>>>>> interfaces. Like most things in cassandra / cloud, the only way to >>>>>>>>> know for >>>>>>>>> sure is to test it yourself and see if observed latency is acceptable >>>>>>>>> (or >>>>>>>>> trust our testing, if you assume we’re sufficiently smart and honest). >>>>>>>>> >>>>>>>>> > Did your test use Amazon EBS–Optimized Instances? >>>>>>>>> >>>>>>>>> We tested dozens of instance type/size combinations (literally). >>>>>>>>> The best performance was clearly with ebs-optimized instances that >>>>>>>>> also >>>>>>>>> have enhanced networking (c4, m4, etc) - slide 43 >>>>>>>>> >>>>>>>>> > SSD or magnetic or does it make any difference? >>>>>>>>> >>>>>>>>> SSD, GP2 (slide 64) >>>>>>>>> >>>>>>>>> > What info is available on EBS performance at peak times, when >>>>>>>>> multiple AWS customers have spikes of demand? >>>>>>>>> >>>>>>>>> Not published, but experiments show that we can hit 10k iops all >>>>>>>>> day every day with only trivial noisy neighbor problems, not enough to >>>>>>>>> impact a real cluster (slide 58) >>>>>>>>> >>>>>>>>> > Is RAID much of a factor or help at all using EBS? >>>>>>>>> >>>>>>>>> You can use RAID to get higher IOPS than you’d normally get by >>>>>>>>> default (GP2 IOPS cap is 10k, which you get with a 3.333T volume – if >>>>>>>>> you >>>>>>>>> need more than 10k, you can stripe volumes together up to the ebs >>>>>>>>> network >>>>>>>>> link max) (hinted at in slide 64) >>>>>>>>> >>>>>>>>> > How exactly is EBS provisioned in terms of its own HA - I mean, >>>>>>>>> with a properly configured Cassandra cluster RF provides HA, so what >>>>>>>>> is the >>>>>>>>> equivalent for EBS? If I have RF=3, what assurance is there that those >>>>>>>>> three EBS volumes aren't all in the same physical rack? >>>>>>>>> >>>>>>>>> There is HA, I’m not sure that AWS publishes specifics. >>>>>>>>> Occasionally specific volumes will have issues (hypervisor’s dedicated >>>>>>>>> ethernet link to EBS network fails, for example). Occasionally >>>>>>>>> instances >>>>>>>>> will have issues. The volume-specific issues seem to be less common >>>>>>>>> than >>>>>>>>> the instance-store “instance retired” or “instance is running on >>>>>>>>> degraded >>>>>>>>> hardware” events. Stop/Start and you’ve recovered (possible with EBS, >>>>>>>>> not >>>>>>>>> possible with instance store). The assurances are in AWS’ SLA – if >>>>>>>>> the SLA >>>>>>>>> is insufficient (and it probably is insufficient), use more than one >>>>>>>>> AZ >>>>>>>>> and/or AWS region or cloud vendor. >>>>>>>>> >>>>>>>>> > For multi-data center operation, what configuration options >>>>>>>>> assure that the EBS volumes for each DC are truly physically >>>>>>>>> separated? >>>>>>>>> >>>>>>>>> It used to be true that EBS control plane for a given region >>>>>>>>> spanned AZs. That’s no longer true. AWS asserts that failure modes >>>>>>>>> for each >>>>>>>>> AZ are isolated (data may replicate between AZs, but a full outage in >>>>>>>>> us-east-1a shouldn’t affect running ebs volumes in us-east-1b or >>>>>>>>> us-east-1c). Slide 65 >>>>>>>>> >>>>>>>>> > In terms of syncing data for the commit log, if the OS call to >>>>>>>>> sync an EBS volume returns, is the commit log data absolutely 100% >>>>>>>>> synced >>>>>>>>> at the hardware level on the EBS end, such that a power failure of the >>>>>>>>> systems on which the EBS volumes reside will still guarantee >>>>>>>>> availability >>>>>>>>> of the fsynced data. As well, is return from fsync an absolute >>>>>>>>> guarantee of >>>>>>>>> sstable durability when Cassandra is about to delete the commit log, >>>>>>>>> including when the two are on different volumes? In practice, we >>>>>>>>> would like >>>>>>>>> some significant degree of pipelining of data, such as during the full >>>>>>>>> processing of flushing memtables, but for the fsync at the end a solid >>>>>>>>> guarantee is needed. >>>>>>>>> >>>>>>>>> Most of the answers in this block are “probably not 100%, you >>>>>>>>> should be writing to more than one host/AZ/DC/vendor to protect your >>>>>>>>> organization from failures”. AWS targets something like 0.1% annual >>>>>>>>> failure >>>>>>>>> rate per volume and 99.999% availability (slide 66). We believe >>>>>>>>> they’re >>>>>>>>> exceeding those goals (at least based with the petabytes of data we >>>>>>>>> have on >>>>>>>>> gp2 volumes). >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> From: Jack Krupansky >>>>>>>>> Reply-To: "user@cassandra.apache.org" >>>>>>>>> Date: Monday, February 1, 2016 at 5:51 AM >>>>>>>>> >>>>>>>>> To: "user@cassandra.apache.org" >>>>>>>>> Subject: Re: EC2 storage options for C* >>>>>>>>> >>>>>>>>> I'm not a fan of guy - this appears to be the slideshare >>>>>>>>> corresponding to the video: >>>>>>>>> >>>>>>>>> http://www.slideshare.net/AmazonWebServices/bdt323-amazon-ebs-cassandra-1-million-writes-per-second >>>>>>>>> >>>>>>>>> My apologies if my questions are actually answered on the video or >>>>>>>>> slides, I just did a quick scan of the slide text. >>>>>>>>> >>>>>>>>> I'm curious where the EBS physical devices actually reside - are >>>>>>>>> they in the same rack, the same data center, same availability zone? I >>>>>>>>> mean, people try to minimize network latency between nodes, so how >>>>>>>>> exactly >>>>>>>>> is EBS able to avoid network latency? >>>>>>>>> >>>>>>>>> Did your test use Amazon EBS–Optimized Instances? >>>>>>>>> >>>>>>>>> SSD or magnetic or does it make any difference? >>>>>>>>> >>>>>>>>> What info is available on EBS performance at peak times, when >>>>>>>>> multiple AWS customers have spikes of demand? >>>>>>>>> >>>>>>>>> Is RAID much of a factor or help at all using EBS? >>>>>>>>> >>>>>>>>> How exactly is EBS provisioned in terms of its own HA - I mean, >>>>>>>>> with a properly configured Cassandra cluster RF provides HA, so what >>>>>>>>> is the >>>>>>>>> equivalent for EBS? If I have RF=3, what assurance is there that those >>>>>>>>> three EBS volumes aren't all in the same physical rack? >>>>>>>>> >>>>>>>>> For multi-data center operation, what configuration options assure >>>>>>>>> that the EBS volumes for each DC are truly physically separated? >>>>>>>>> >>>>>>>>> In terms of syncing data for the commit log, if the OS call to >>>>>>>>> sync an EBS volume returns, is the commit log data absolutely 100% >>>>>>>>> synced >>>>>>>>> at the hardware level on the EBS end, such that a power failure of the >>>>>>>>> systems on which the EBS volumes reside will still guarantee >>>>>>>>> availability >>>>>>>>> of the fsynced data. As well, is return from fsync an absolute >>>>>>>>> guarantee of >>>>>>>>> sstable durability when Cassandra is about to delete the commit log, >>>>>>>>> including when the two are on different volumes? In practice, we >>>>>>>>> would like >>>>>>>>> some significant degree of pipelining of data, such as during the full >>>>>>>>> processing of flushing memtables, but for the fsync at the end a solid >>>>>>>>> guarantee is needed. >>>>>>>>> >>>>>>>>> >>>>>>>>> -- Jack Krupansky >>>>>>>>> >>>>>>>>> On Mon, Feb 1, 2016 at 12:56 AM, Eric Plowe <eric.pl...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Jeff, >>>>>>>>>> >>>>>>>>>> If EBS goes down, then EBS Gp2 will go down as well, no? I'm not >>>>>>>>>> discounting EBS, but prior outages are worrisome. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Sunday, January 31, 2016, Jeff Jirsa < >>>>>>>>>> jeff.ji...@crowdstrike.com> wrote: >>>>>>>>>> >>>>>>>>>>> Free to choose what you'd like, but EBS outages were also >>>>>>>>>>> addressed in that video (second half, discussion by Dennis Opacki). >>>>>>>>>>> 2016 >>>>>>>>>>> EBS isn't the same as 2011 EBS. >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Jeff Jirsa >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Jan 31, 2016, at 8:27 PM, Eric Plowe <eric.pl...@gmail.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Thank you all for the suggestions. I'm torn between GP2 vs >>>>>>>>>>> Ephemeral. GP2 after testing is a viable contender for our >>>>>>>>>>> workload. The >>>>>>>>>>> only worry I have is EBS outages, which have happened. >>>>>>>>>>> >>>>>>>>>>> On Sunday, January 31, 2016, Jeff Jirsa < >>>>>>>>>>> jeff.ji...@crowdstrike.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Also in that video - it's long but worth watching >>>>>>>>>>>> >>>>>>>>>>>> We tested up to 1M reads/second as well, blowing out page cache >>>>>>>>>>>> to ensure we weren't "just" reading from memory >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Jeff Jirsa >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Jan 31, 2016, at 9:52 AM, Jack Krupansky < >>>>>>>>>>>> jack.krupan...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>> How about reads? Any differences between read-intensive and >>>>>>>>>>>> write-intensive workloads? >>>>>>>>>>>> >>>>>>>>>>>> -- Jack Krupansky >>>>>>>>>>>> >>>>>>>>>>>> On Sun, Jan 31, 2016 at 3:13 AM, Jeff Jirsa < >>>>>>>>>>>> jeff.ji...@crowdstrike.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi John, >>>>>>>>>>>>> >>>>>>>>>>>>> We run using 4T GP2 volumes, which guarantee 10k iops. Even at >>>>>>>>>>>>> 1M writes per second on 60 nodes, we didn’t come close to hitting >>>>>>>>>>>>> even 50% >>>>>>>>>>>>> utilization (10k is more than enough for most workloads). PIOPS >>>>>>>>>>>>> is not >>>>>>>>>>>>> necessary. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> From: John Wong >>>>>>>>>>>>> Reply-To: "user@cassandra.apache.org" >>>>>>>>>>>>> Date: Saturday, January 30, 2016 at 3:07 PM >>>>>>>>>>>>> To: "user@cassandra.apache.org" >>>>>>>>>>>>> Subject: Re: EC2 storage options for C* >>>>>>>>>>>>> >>>>>>>>>>>>> For production I'd stick with ephemeral disks (aka instance >>>>>>>>>>>>> storage) if you have running a lot of transaction. >>>>>>>>>>>>> However, for regular small testing/qa cluster, or something >>>>>>>>>>>>> you know you want to reload often, EBS is definitely good enough >>>>>>>>>>>>> and we >>>>>>>>>>>>> haven't had issues 99%. The 1% is kind of anomaly where we have >>>>>>>>>>>>> flush >>>>>>>>>>>>> blocked. >>>>>>>>>>>>> >>>>>>>>>>>>> But Jeff, kudo that you are able to use EBS. I didn't go >>>>>>>>>>>>> through the video, do you actually use PIOPS or just standard GP2 >>>>>>>>>>>>> in your >>>>>>>>>>>>> production cluster? >>>>>>>>>>>>> >>>>>>>>>>>>> On Sat, Jan 30, 2016 at 1:28 PM, Bryan Cheng < >>>>>>>>>>>>> br...@blockcypher.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Yep, that motivated my question "Do you have any idea what >>>>>>>>>>>>>> kind of disk performance you need?". If you need the >>>>>>>>>>>>>> performance, its hard >>>>>>>>>>>>>> to beat ephemeral SSD in RAID 0 on EC2, and its a solid, battle >>>>>>>>>>>>>> tested >>>>>>>>>>>>>> configuration. If you don't, though, EBS GP2 will save a _lot_ >>>>>>>>>>>>>> of headache. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Personally, on small clusters like ours (12 nodes), we've >>>>>>>>>>>>>> found our choice of instance dictated much more by the balance >>>>>>>>>>>>>> of price, >>>>>>>>>>>>>> CPU, and memory. We're using GP2 SSD and we find that for our >>>>>>>>>>>>>> patterns the >>>>>>>>>>>>>> disk is rarely the bottleneck. YMMV, of course. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, Jan 29, 2016 at 7:32 PM, Jeff Jirsa < >>>>>>>>>>>>>> jeff.ji...@crowdstrike.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> If you have to ask that question, I strongly recommend m4 or >>>>>>>>>>>>>>> c4 instances with GP2 EBS. When you don’t care about replacing >>>>>>>>>>>>>>> a node >>>>>>>>>>>>>>> because of an instance failure, go with i2+ephemerals. Until >>>>>>>>>>>>>>> then, GP2 EBS >>>>>>>>>>>>>>> is capable of amazing things, and greatly simplifies life. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We gave a talk on this topic at both Cassandra Summit and >>>>>>>>>>>>>>> AWS re:Invent: https://www.youtube.com/watch?v=1R-mgOcOSd4 It’s >>>>>>>>>>>>>>> very much a viable option, despite any old documents online >>>>>>>>>>>>>>> that say >>>>>>>>>>>>>>> otherwise. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> From: Eric Plowe >>>>>>>>>>>>>>> Reply-To: "user@cassandra.apache.org" >>>>>>>>>>>>>>> Date: Friday, January 29, 2016 at 4:33 PM >>>>>>>>>>>>>>> To: "user@cassandra.apache.org" >>>>>>>>>>>>>>> Subject: EC2 storage options for C* >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> My company is planning on rolling out a C* cluster in EC2. >>>>>>>>>>>>>>> We are thinking about going with ephemeral SSDs. The question is >>>>>>>>>>>>>>> this: Should we put two in RAID 0 or just go with one? We >>>>>>>>>>>>>>> currently run a >>>>>>>>>>>>>>> cluster in our data center with 2 250gig Samsung 850 EVO's in >>>>>>>>>>>>>>> RAID 0 and we >>>>>>>>>>>>>>> are happy with the performance we are seeing thus far. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks! >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Eric >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Steve Robenalt >>>>>>> Software Architect >>>>>>> sroben...@highwire.org <bza...@highwire.org> >>>>>>> (office/cell): 916-505-1785 >>>>>>> >>>>>>> HighWire Press, Inc. >>>>>>> 425 Broadway St, Redwood City, CA 94063 >>>>>>> www.highwire.org >>>>>>> >>>>>>> Technology for Scholarly Communication >>>>>>> >>>>>> >>>>>> >>>>> -- >>>> Ben Bromhead >>>> CTO | Instaclustr >>>> +1 650 284 9692 >>>> >>> >>> >> >
