Re: Need help with simple schema for time-series

[Noorul Islam K M]

http://kairosdb.github.io/

Regards,
Noorul

Peter Figliozzi  writes:

> I have data from many sensors as time-series:
>
>- Sensor name
>- Date
>- Time
>- value
>
> I want to query windows of both date and time.  For example, 8am - 9am from
> Aug. 1st to Aug 10th.
>
> Here's what I did:
>
> CREATE TABLE mykeyspace.mytable (
> sensorname text,
> date date,
> time time,
> data MAP,
> PRIMARY KEY (sensorname, date, time)
> );
>
>
> However, when we query this, Cassandra restricts us to an "equal" relation
> for the date, if we are to select a window of time.  So with that schema,
> I'd have to query once for each date.
>
>
> What's the right way to do this??  ("Right" defined as extracting a window
> of date and of time in one query.)
>
>
> Thank you,
>
>
> Pete

Re: Question about hector api documentation

[Noorul Islam K M]

The very first line README tells the story


THIS PROJECT IS NO LONGER ACTIVE


But you should be able to generate doc from source code.

Regards,
Noorul


Sungju Hong  writes:

> Hello,
>
> I'm finding hector java api doc.
>
> I searched though google but couldn't find hector api doc.
>
> This link is broken also.
> https://hector-client.github.io/hector/build/html/content/api.html#
>
> Can I know the way to get the doc?
>
> Thanks.
>
> Sungju.

Re: Too many sstables with DateTieredCompactionStrategy

[Noorul Islam K M]

Lyubo Kamenov <k...@heroku.com> writes:

> Maybe increase the number of tables that can be compacted by minor
> compactions[1],
> i.e. max_threshold (default is set to 32).
>
> 1.
> https://docs.datastax.com/en/cql/3.1/cql/cql_reference/compactSubprop.html?scroll=compactSubprop__compactionSubpropertiesDTCS
>

I see that after the timestamp_resolution change to MICROSECONDS, slowly
the count is decreasing. I will give it some more time and if not
helping I will try putting higher value for max_threshold.

Thanks and Regards
Noorul

> On Mon, Feb 29, 2016 at 9:28 PM, Noorul Islam Kamal Malmiyoda <
> noo...@noorul.com> wrote:
>
>> Hello Marcus,
>>
>> I altered the table to set timestamp_resolution to 'MICROSECONDS'. I
>> waited for sometime, but the sstable count did not come down. Do you
>> think I should specific command to reduce the count of sstables after
>> setting this?
>>
>> Thanks and Regards
>> Noorul
>>
>>
>> On Mon, Feb 29, 2016 at 7:22 PM, Marcus Eriksson <krum...@gmail.com>
>> wrote:
>> > why do you have 'timestamp_resolution': 'MILLISECONDS'? It should be
>> left as
>> > default (MICROSECONDS) unless you do "USING TIMESTAMP
>> > "-inserts, see
>> > https://issues.apache.org/jira/browse/CASSANDRA-11041
>> >
>> > On Mon, Feb 29, 2016 at 2:36 PM, Noorul Islam K M <noo...@noorul.com>
>> wrote:
>> >>
>> >>
>> >> Hi all,
>> >>
>> >> We are using below compaction settings for a table
>> >>
>> >> compaction = {'timestamp_resolution': 'MILLISECONDS',
>> >> 'max_sstable_age_days': '365', 'base_time_seconds': '60', 'class':
>> >> 'org.apache.cassandra.db.compaction.DateTieredCompactionStrategy'}
>> >>
>> >> But it is creating too many sstables. Currently number of sstables
>> >> is 4. We have been injecting data for the last three days.
>> >>
>> >> We have set the compactionthroughput to 128 MB/s
>> >>
>> >> $ nodetool getcompactionthroughput
>> >>
>> >> Current compaction throughput: 128 MB/s
>> >>
>> >> But this is not helping.
>> >>
>> >> How can we control the number of sstables in this case?
>> >>
>> >> Thanks and Regards
>> >> Noorul
>> >
>> >
>>

Re: Too many sstables with DateTieredCompactionStrategy

[Noorul Islam K M]

Alain RODRIGUEZ <arodr...@gmail.com> writes:

> Might be due to this:
>
> Fixed in 2.1.12 (Assuming you are using C*2.1):
> https://issues.apache.org/jira/browse/CASSANDRA-10422
>
> Some question to have more context:
>
>
>1. What C* version are you using?

We are using DSE 4.8.3, hence Apache Cassandra 2.1.12.1046

>2. Do you use vnodes?

Yes

>3. How many vnodes per node?

32

>4. How many nodes / DC do you have?

We have 3 DCs

cassandra, spark and solr.

The keyspace has RF 3 in cassandra and solr DC.


>5. How do you run repairs (tool & command)?

We enabled auto repair feature provided by OpsCenter.

Thanks and Regards
Noorul


> C*heers,
>
> ---
> Alain Rodriguez - al...@thelastpickle.com
> France
>
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> 2016-02-29 15:50 GMT+01:00 Noorul Islam Kamal Malmiyoda <noo...@noorul.com>:
>
>> Yes, we have enabled it on OpsCenter. Is that the reason?
>> On Feb 29, 2016 8:07 PM, "Dominik Keil" <dominik.k...@movilizer.com>
>> wrote:
>>
>>> Are you using incremental repais?
>>>
>>> Am 29.02.2016 um 14:36 schrieb Noorul Islam K M:
>>>
>>> Hi all,
>>>
>>> We are using below compaction settings for a table
>>>
>>> compaction = {'timestamp_resolution': 'MILLISECONDS',
>>> 'max_sstable_age_days': '365', 'base_time_seconds': '60', 'class':
>>> 'org.apache.cassandra.db.compaction.DateTieredCompactionStrategy'}
>>>
>>> But it is creating too many sstables. Currently number of sstables
>>> is 4. We have been injecting data for the last three days.
>>>
>>> We have set the compactionthroughput to 128 MB/s
>>>
>>> $ nodetool getcompactionthroughput
>>>
>>> Current compaction throughput: 128 MB/s
>>>
>>> But this is not helping.
>>>
>>> How can we control the number of sstables in this case?
>>>
>>> Thanks and Regards
>>> Noorul
>>>
>>>
>>> --
>>> *Dominik Keil*
>>> Phone: + 49 (0) 621 150 207 31
>>> Mobile: + 49 (0) 151 626 602 14
>>>
>>> Movilizer GmbH
>>> Julius-Hatry-Strasse 1
>>> 68163 Mannheim
>>> Germany
>>>
>>> movilizer.com
>>>
>>> [image: Visit company website] <http://movilizer.com/>
>>> *Reinvent Your Mobile Enterprise*
>>>
>>> <http://movilizer.com/training>
>>> <http://movilizer.com/training>
>>>
>>> *Be the first to know:*
>>> Twitter <https://twitter.com/Movilizer> | LinkedIn
>>> <https://www.linkedin.com/company/movilizer-gmbh> | Facebook
>>> <https://www.facebook.com/Movilizer> | stack overflow
>>> <http://stackoverflow.com/questions/tagged/movilizer>
>>>
>>> Company's registered office: Mannheim HRB: 700323 / Country Court:
>>> Mannheim Managing Directors: Alberto Zamora, Jörg Bernauer, Oliver Lesche
>>> Please inform us immediately if this e-mail and/or any attachment was
>>> transmitted incompletely or was not intelligible.
>>>
>>> This e-mail and any attachment is for authorized use by the intended
>>> recipient(s) only. It may contain proprietary material, confidential
>>> information and/or be subject to legal privilege. It should not be
>>> copied, disclosed to, retained or used by any other party. If you are not
>>> an intended recipient then please promptly delete this e-mail and any
>>> attachment and all copies and inform the sender.
>>
>>

Too many sstables with DateTieredCompactionStrategy

[Noorul Islam K M]

Hi all,

We are using below compaction settings for a table

compaction = {'timestamp_resolution': 'MILLISECONDS',
'max_sstable_age_days': '365', 'base_time_seconds': '60', 'class':
'org.apache.cassandra.db.compaction.DateTieredCompactionStrategy'}

But it is creating too many sstables. Currently number of sstables
is 4. We have been injecting data for the last three days.

We have set the compactionthroughput to 128 MB/s

$ nodetool getcompactionthroughput

Current compaction throughput: 128 MB/s

But this is not helping. 

How can we control the number of sstables in this case?

Thanks and Regards
Noorul

sstableloader throughput

[Noorul Islam K M]

I have a need to stream data to new cluster using sstableloader. I
spawned a machine with 32 cores assuming that sstableloader scaled with
respect to cores. But it doesn't look like so.

I am getting an average throughput of 18 MB/s which seems to be pretty
low (I might be wrong).

Is there any way to increase the throughput. OpsCenter data on target
cluster shows very less write requests / second.

Thanks and Regards
Noorul

What is the ideal way to merge two Cassandra clusters with same keyspace into one?

[Noorul Islam K M]

Hello all,

We have two clusters X and Y with same keyspaces but distinct data sets.
We are planning to merge these into single cluster. What would be ideal
steps to achieve this without downtime for applications? We have time
series data stream continuously writing to Cassandra.

We have ruled out export/import as that will make us loose data during
the time of copy.

We also ruled out sstableloader as that is not reliable. It fails often
and there is not way to start from where it failed.

Any suggestions will help.

Thanks and Regards
Noorul

Re: What is the ideal way to merge two Cassandra clusters with same keyspace into one?

[Noorul Islam K M]

George Sigletos <sigle...@textkernel.nl> writes:

> Hello,
>
> We had a similar problem where we needed to migrate data from one cluster
> to another.
>
> We ended up using Spark to accomplish this. It is fast and reliable but
> some downtime was required after all.
>
> We minimized the downtime by doing a first run, and then run incremental
> updates.
>

How much data are you talking about?

How did you achieve incremental run? We are using kairosdb and some of
the other schemas does not have a way to filter based on date.

Thanks and Regards
Noorul

> Kind regards,
> George
>
>
>
> On Mon, Dec 21, 2015 at 10:12 AM, Noorul Islam K M <noo...@noorul.com>
> wrote:
>
>>
>> Hello all,
>>
>> We have two clusters X and Y with same keyspaces but distinct data sets.
>> We are planning to merge these into single cluster. What would be ideal
>> steps to achieve this without downtime for applications? We have time
>> series data stream continuously writing to Cassandra.
>>
>> We have ruled out export/import as that will make us loose data during
>> the time of copy.
>>
>> We also ruled out sstableloader as that is not reliable. It fails often
>> and there is not way to start from where it failed.
>>
>> Any suggestions will help.
>>
>> Thanks and Regards
>> Noorul
>>