[tor-bugs] #24554 [Core Tor/Tor]: sched: Have per-scheduler type data in a channel_t

[Tor Bug Tracker & Wiki]

#24554: sched: Have per-scheduler type data in a channel_t
--+
 Reporter:  dgoulet   |  Owner:  dgoulet
 Type:  enhancement   | Status:  assigned
 Priority:  Medium|  Milestone:  Tor: 0.3.3.x-final
Component:  Core Tor/Tor  |Version:
 Severity:  Normal|   Keywords:  tor-sched
Actual Points:|  Parent ID:
   Points:|   Reviewer:
  Sponsor:|
--+
 Right now in `channel_t`, we have a `sched_heap_idx` and
 `scheduler_state`. Both are specific to the scheduler layer and are per-
 channel. The KIST scheduler also keeps data per-channel in a global state
 it keeps some other place.

 Because of #23744, which is a ticket about the scheduler state not
 applying to both Vanilla and KIST, we need a way to have a interface that
 allows any "scheduler data per-channel" to be specific to a type of
 scheduler (KIST or Vanilla).

 To achieve compartmentalization of whatever state the scheduler needs to
 keep, we need an interface that allows a specific scheduler to store data
 per-channel instead of in a global state which, in the case of KIST, would
 avoid two things:

 1. Huge amount of memory allocation at runtime that needs to be done by
 the main loop (this is currently a problem with KIST code).

 2. Avoid data duplication or/and synchronization problem between the
 channel subsystem and the scheduler subsystem.

 Finally, by doing so, we can build an opaque interface for the specific
 scheduler information data structure in order to make them *ONLY*
 accessible from the per-type scheduler code. This means a builtin
 protection to prevent any other layer in the tor to change or access the
 channel's scheduler state.

 Once we have this capability, we can go ahead and fix #23744 and move the
 `sched_heap_idx` and `scheduler_state` out of a channel object so ONLY the
 scheduler has access to those and be able to have different semantic for
 different scheduler type.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #24554 [Core Tor/Tor]: sched: Have per-scheduler type data in a channel_t

[Tor Bug Tracker & Wiki]

#24554: sched: Have per-scheduler type data in a channel_t
-+-
 Reporter:  dgoulet  |  Owner:  (none)
 Type:  enhancement  | Status:
 |  assigned
 Priority:  Very High|  Milestone:  Tor:
 |  unspecified
Component:  Core Tor/Tor |Version:
 Severity:  Normal   | Resolution:
 Keywords:  tor-sched, 034-triage-20180328,  |  Actual Points:
  034-removed-20180328, tor-chan |
Parent ID:  #23993   | Points:
 Reviewer:   |Sponsor:
-+-
Changes (by dgoulet):

 * keywords:  tor-sched, 034-triage-20180328, 034-removed-20180328 => tor-
 sched, 034-triage-20180328, 034-removed-20180328, tor-chan


Comment:

 Branch `ticket24554_033_02` has all the things! Wow.

 It is way too big. It should be broken down into smaller pieces (tickets)
 and then properly tested.

 This should really go in tor because it will help greatly to reduce our
 technical debt around the scheduler subsystem. But also, one particular
 thing, is that it removes the heap memory allocation done for each channel
 at each KIST mainloop. Under pressure, this leads to memory fragmentation
 quite a bit, and it has been observed.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #24554 [Core Tor/Tor]: sched: Have per-scheduler type data in a channel_t

[Tor Bug Tracker & Wiki]

#24554: sched: Have per-scheduler type data in a channel_t
-+-
 Reporter:  dgoulet  |  Owner:  (none)
 Type:  enhancement  | Status:
 |  assigned
 Priority:  Very High|  Milestone:  Tor:
 |  unspecified
Component:  Core Tor/Tor |Version:
 Severity:  Normal   | Resolution:
 Keywords:  tor-sched, 034-triage-20180328,  |  Actual Points:
  034-removed-20180328, tor-chan, network-team-  |
  roadmap-2020Q1 |
Parent ID:  #23993   | Points:
 Reviewer:   |Sponsor:
-+-
Changes (by gaba):

 * keywords:  tor-sched, 034-triage-20180328, 034-removed-20180328, tor-chan
 =>
 tor-sched, 034-triage-20180328, 034-removed-20180328, tor-chan,
 network-team-roadmap-2020Q1


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #24554 [Core Tor/Tor]: sched: Have per-scheduler type data in a channel_t

[Tor Bug Tracker & Wiki]

#24554: sched: Have per-scheduler type data in a channel_t
--+
 Reporter:  dgoulet   |  Owner:  dgoulet
 Type:  enhancement   | Status:  needs_review
 Priority:  Medium|  Milestone:  Tor: 0.3.3.x-final
Component:  Core Tor/Tor  |Version:
 Severity:  Normal| Resolution:
 Keywords:  tor-sched |  Actual Points:
Parent ID:| Points:
 Reviewer:|Sponsor:
--+
Changes (by dgoulet):

 * status:  assigned => needs_review


Comment:

 See branch: `ticket24554_033_01`
 OnionGit: https://oniongit.eu/dgoulet/tor/merge_requests/12

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #24554 [Core Tor/Tor]: sched: Have per-scheduler type data in a channel_t

[Tor Bug Tracker & Wiki]

#24554: sched: Have per-scheduler type data in a channel_t
--+
 Reporter:  dgoulet   |  Owner:  dgoulet
 Type:  enhancement   | Status:  needs_review
 Priority:  Medium|  Milestone:  Tor: 0.3.3.x-final
Component:  Core Tor/Tor  |Version:
 Severity:  Normal| Resolution:
 Keywords:  tor-sched |  Actual Points:
Parent ID:  #23993| Points:
 Reviewer:|Sponsor:
--+
Changes (by dgoulet):

 * parent:   => #23993


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #24554 [Core Tor/Tor]: sched: Have per-scheduler type data in a channel_t

[Tor Bug Tracker & Wiki]

#24554: sched: Have per-scheduler type data in a channel_t
--+
 Reporter:  dgoulet   |  Owner:  dgoulet
 Type:  enhancement   | Status:  needs_review
 Priority:  Medium|  Milestone:  Tor: 0.3.3.x-final
Component:  Core Tor/Tor  |Version:
 Severity:  Normal| Resolution:
 Keywords:  tor-sched |  Actual Points:
Parent ID:  #23993| Points:
 Reviewer:  pastly|Sponsor:
--+
Changes (by pastly):

 * reviewer:   => pastly


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #24554 [Core Tor/Tor]: sched: Have per-scheduler type data in a channel_t

[Tor Bug Tracker & Wiki]

#24554: sched: Have per-scheduler type data in a channel_t
--+
 Reporter:  dgoulet   |  Owner:  dgoulet
 Type:  enhancement   | Status:  needs_revision
 Priority:  Medium|  Milestone:  Tor: 0.3.3.x-final
Component:  Core Tor/Tor  |Version:
 Severity:  Normal| Resolution:
 Keywords:  tor-sched |  Actual Points:
Parent ID:  #23993| Points:
 Reviewer:  pastly|Sponsor:
--+
Changes (by pastly):

 * status:  needs_review => needs_revision


Comment:

 Left comments on oniongit.

 - One easy to fix compile issue.
 - Some rambling thoughts about the consequences of ignoring a `chan` with
 invalid `sched_info` in the middle of a scheduling run.

 Marking needs_revision for the compile issue.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #24554 [Core Tor/Tor]: sched: Have per-scheduler type data in a channel_t

[Tor Bug Tracker & Wiki]

#24554: sched: Have per-scheduler type data in a channel_t
--+
 Reporter:  dgoulet   |  Owner:  dgoulet
 Type:  enhancement   | Status:  needs_revision
 Priority:  Medium|  Milestone:  Tor: 0.3.3.x-final
Component:  Core Tor/Tor  |Version:
 Severity:  Normal| Resolution:
 Keywords:  tor-sched |  Actual Points:
Parent ID:  #23993| Points:
 Reviewer:  pastly|Sponsor:
--+

Comment (by dgoulet):

 pastly did review and I fixed stuff. I'll rebase this on latest master and
 open a new merge request for upstream merge.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #24554 [Core Tor/Tor]: sched: Have per-scheduler type data in a channel_t

[Tor Bug Tracker & Wiki]

#24554: sched: Have per-scheduler type data in a channel_t
--+
 Reporter:  dgoulet   |  Owner:  dgoulet
 Type:  enhancement   | Status:  needs_review
 Priority:  Medium|  Milestone:  Tor: 0.3.3.x-final
Component:  Core Tor/Tor  |Version:
 Severity:  Normal| Resolution:
 Keywords:  tor-sched |  Actual Points:
Parent ID:  #23993| Points:
 Reviewer:  pastly|Sponsor:
--+
Changes (by dgoulet):

 * status:  needs_revision => needs_review


Comment:

 See branch: `ticket24554_033_02`.
 Oniongit MR: https://oniongit.eu/dgoulet/tor/merge_requests/13

 One thing to notice is that the top commit is actually the commit for
 #23579. It ain't very complicated, removes code duplication and lays the
 ground work for future tickets like #23744 and the work to move the
 scheduler state from the `channel_t` object into `sched_info_t` *only* for
 Vanilla because KIST doesn't need it.

 This is a slow and complicated work towards getting the scheduler
 subsystem much simpler, robust and easier to test. All this is part of
 #23993 parent ticket.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #24554 [Core Tor/Tor]: sched: Have per-scheduler type data in a channel_t

[Tor Bug Tracker & Wiki]

#24554: sched: Have per-scheduler type data in a channel_t
--+
 Reporter:  dgoulet   |  Owner:  dgoulet
 Type:  enhancement   | Status:  needs_review
 Priority:  Medium|  Milestone:  Tor: 0.3.3.x-final
Component:  Core Tor/Tor  |Version:
 Severity:  Normal| Resolution:
 Keywords:  tor-sched |  Actual Points:
Parent ID:  #23993| Points:
 Reviewer:|Sponsor:
--+
Changes (by dgoulet):

 * reviewer:  pastly =>


Comment:

 (Removing pastly since he did a first review and might not want to do a
 second ;)

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #24554 [Core Tor/Tor]: sched: Have per-scheduler type data in a channel_t

[Tor Bug Tracker & Wiki]

#24554: sched: Have per-scheduler type data in a channel_t
--+
 Reporter:  dgoulet   |  Owner:  dgoulet
 Type:  enhancement   | Status:  needs_review
 Priority:  Medium|  Milestone:  Tor: 0.3.3.x-final
Component:  Core Tor/Tor  |Version:
 Severity:  Normal| Resolution:
 Keywords:  tor-sched |  Actual Points:
Parent ID:  #23993| Points:
 Reviewer:|Sponsor:
--+

Comment (by dgoulet):

 Quick note. Moving `sched_heap_idx` out of `channel_t` and into
 `sched_info_t` is a bit more difficult because we use this `offsetof()`
 strategy with priority list so `sched_heap_idx` needs to offset from the
 `channel_t` object which is what is stored in the list. One possible
 solution would be to keep a pointer to the channel inside the
 `sched_info_t` and store those objects instead.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #24554 [Core Tor/Tor]: sched: Have per-scheduler type data in a channel_t

[Tor Bug Tracker & Wiki]

#24554: sched: Have per-scheduler type data in a channel_t
--+
 Reporter:  dgoulet   |  Owner:  dgoulet
 Type:  enhancement   | Status:  needs_revision
 Priority:  Medium|  Milestone:  Tor: 0.3.3.x-final
Component:  Core Tor/Tor  |Version:
 Severity:  Normal| Resolution:
 Keywords:  tor-sched |  Actual Points:
Parent ID:  #23993| Points:
 Reviewer:|Sponsor:
--+
Changes (by dgoulet):

 * status:  needs_review => needs_revision


Comment:

 I need to run this a bit more on top of #24665 and #24666.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #24554 [Core Tor/Tor]: sched: Have per-scheduler type data in a channel_t

[Tor Bug Tracker & Wiki]

#24554: sched: Have per-scheduler type data in a channel_t
--+
 Reporter:  dgoulet   |  Owner:  dgoulet
 Type:  enhancement   | Status:  needs_revision
 Priority:  Very High |  Milestone:  Tor: 0.3.4.x-final
Component:  Core Tor/Tor  |Version:
 Severity:  Normal| Resolution:
 Keywords:  tor-sched |  Actual Points:
Parent ID:  #23993| Points:
 Reviewer:|Sponsor:
--+
Changes (by dgoulet):

 * milestone:  Tor: 0.3.3.x-final => Tor: 0.3.4.x-final


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs