[Proposal Idea] Incentivized Testnet Proposal: Casper Network Testnet Rewards Program (Oct 1 - Dec 31, 2024)

Proposal Discussion Proposal Ideas
1

Friendly Introduction

Dear Testnet Validators:

We realize that we are approaching the end of the Q3 Testnet phase, and the beginning of the 4th quarter, without a formalized new program in place. I personally apologize for not publishing this proposal sooner, and hope you understand this in the context of the many other items on my plate, and not view it as a lack of interest in, prioritization of or respect for the Testnet program participants. That said, below is the Casper Association’s formal proposal for the next phase(s) of the Casper Testnet. As I promised, the community and stakeholder going forward are intimately involved in evaluating program proposals, amongst other things, and this is no exception.
There are certain constraints that I will enumerate below, but first let me list what we tried to prioritize, based on the Testnet Validator’s prior feedback:

  1. A Testnet that more closely resembles an aspirational version of Mainnet, with a healthy mix of TTL and Genesis nodes, validating and non-validating.
  2. No more “black box” performance tracking - the only performance metric that will be used is entirely on-chain, and verifiable
  3. Fair and predictable compensation
  4. No more Sybil’s
  5. Fair access to participation
  6. Better enable one of the primary use cases of Testnet: allowing mainnet validators to test their set-ups, get access to releases prior to mainnet activation and optimize their mainnet service in a “risk-free” environment

While many of the details of the proposal are up for debate, I ask that the discussion respects the following constraints:

  1. The budget is set.
  2. The quarterly cadence is set
  3. While the protocol is permissionless, the Testnet Program is not. This proposal provides for equal opportunity of access to anyone, but doesn’t guarantee access as an inalienable right to anyone. The number of quarterly participants will be capped at 100.

Finally, I realize that it is the evening of September 26th, and this proposal pertains to the quarter beginning in 4 days. We won’t realistically complete our deliberations on this proposal in the next 4 days, and I expect we will kick off the registration process a few days into October. While I would hope that the current participants can be somewhat flexible with the few days of uncertainty, I’m certainly open to suggestions in the comments on how to address the 1-2 weeks of extension of the Q3 program that is inevitable.

With all that said, I thank you in advance for your consideration of and engagement around the proposal below. The Testnet Validator group is of critical importance to the health of the protocol and the ecosystem, and you are all soldiers at the front of the war on :beetle: bugs! We are grateful for all your years of helping to make Casper Network better and bigger, and look forward to doing so with you for many years to come.

With appreciation,

Michael Steuer
CTO, Casper Association

The Proposal

Overview:

The Casper Network is introducing a Testnet Rewards Program for the quarter from October 1, 2024, to December 31, 2024. This program rewards validators for their contributions to maintaining and improving the Testnet.

Program Structure:

1. Application Process:

Application Deadline: xxx, 2024

Application Platform: Validators must apply via an online form, collecting the following:

  • Name and Surname
  • Email Address
  • Cryptographic Signature (used for email verification)
  • Public Key for the validator
  • Type of Validator Preference:
  • TTL Node (Time to Live)
  • Full Archival Node
  • Any (if the applicant is open to either type)
  • Mainnet Validator Status:
  • Are you a mainnet validator? (Yes/No)
  • If yes, provide your Mainnet Validator Public Key.
  • Are you a Top-30 Mainnet Validator? (Yes/No)
  • If yes, would you like to opt into one of the 15 reserved slots for Top-30 Mainnet Validators?
  • If yes, do you want to receive up to EUR 50 per month, or do you prefer to participate without any incentives? (Yes/No)
  • Basic Questions testing knowledge of official validator documentation.
  • Telegram Username
  • Twitter Profile Link

Lottery Selection:

The lottery will take place on xxxx, 2024, using an online tool or on-chain contract for transparency. A total of 100 validators will be selected.

KYC Verification with Video-Based Liveness Check:

Selected validators will undergo KYC verification, including a video-based liveness check, from xxxx to xxxx, 2024.

2. Incentives and Validator Categories:

Validators will be divided into the following categories:

  • 10 Full History Nodes (Archival Nodes): Node operators maintaining historical data are eligible for EUR 200 per month, provided they give proper responses to automated (daily) checks through the RPC and or the SSE ports for existence and availability of historical data. These nodes will serve as data keepers of the network, and will not be in the active validator set, but will have an active bid to be able to step up as a validator in case of need.
  • 85 TTL (Time to Live) Validators: Validators maintaining standard nodes are eligible for EUR 100 per month with 99% or higher on-chain monthly performance.
  • 15 Slots for Top 30 Mainnet Validators:
    • Reserved for high-performing mainnet validators, eligible for up to EUR 50 per month with 99% or higher on-chain monthly performance.
    • Optional Incentives: Top-30 Mainnet Validators can opt out of receiving rewards and participate purely for the Testnet experience, in order to test and improve their mainnet validator activities.

Reward Distribution:

  • Rewards will be distributed quarterly. Node operators maintaining required performance throughout the quarter can earn up to EUR 600 (Full History Nodes), EUR 300 (TTL Validators), or EUR 150 (Top-30 Mainnet Validators).
  • Partial Rewards: Validators who lose eligibility due to low performance, or full archival node operators due to lacking data availability requirements, may still receive partial rewards at the end of the quarter.
  • No Rewards for Disqualification: Validators disqualified for violating the Testnet Community Code of Conduct or engaging in malicious behavior will receive no partial rewards.
  • Every participant must monitor the health and performance of their nodes at all times, intervene in a timely manner when needed, report issues along with relevant logs to Casper Association, and follow the official announcements in the Casper Testnet Participants group.

3. Validator Requirements:

  • Validators must meet specific technical standards, including open ports and servicing random historical blocks (for Full History Nodes). These requirements will be verified by an automated bot. Performance will be measured entirely on-chain, ensuring fairness and transparency.
  • TTL nodes assigned as validators must keep their gossip ports open, and others closed to the public (with an exception for the Casper monitoring tool). Archival nodes must keep all of their ports open in a limited fashion.
  • All nodes must adhere to the minimum requirements outlined in the official documentation. TTL nodes can have a reduced disk size (500GB with RAID 1).

4. Program Budget:

The total budget for the quarter is EUR ~34,000, designed to support high-quality participation while managing costs.

Public Key Submission Limits:

Validators must meet the following requirements for public key submission:

  • Minimum 3 non-transfer deploys on the Mainnet.
  • At least CSPR 1,000 total balance (staked or liquid) on the Mainnet.

Anti-DoS Measures:

To prevent abuse of the application process, the following anti-DoS measures will be in place:

  1. Cryptographic Signature: Validators will provide a cryptographic signature for email verification to ensure authenticity.
  2. One submission per email on the form with a sign-in requirement.
  3. Knowledge-Based Filtering: Applicants will answer basic questions from the official validator documentation to ensure they have the necessary knowledge to operate a node.
  4. KYC and Liveness Check: All selected validators will undergo KYC verification with a video-based liveness check to confirm their identity.

Timeline:

  • Application Deadline: October 06, 2024
  • Lottery Selection and Announcement: October 07 to October 11, 2024
  • KYC Verification (with liveness check): October 07 to October 15, 2024
  • Incentivised Testnet Operation Period: October 15 to December 31, 2024
  • Reward Distribution: Rewards will be distributed quarterly, starting in January 2025.
2 Likes
3

I will make a detailed comment when I have some time but the first thing I would love to see on the new testnet program is to have AUTOMATED payment on time.

4

As promised in the chat, here is our feedback on the proposal.

1. Application Process:

There are no issues here. Perhaps offer KYB instead of KYC for incorporated parties that wish to verify that way.

2. Incentives and Validator Categories:

  • A mix of history and TTL nodes is good, with historical nodes rewarded more.

  • The budget has been lowered significantly from €162,500(?) to ~€34,000, but the proposal contains A LOT of overhead (to manage the program; this is not free and should be taken into account).

  • The proposal contains NO incentivisation for specific tasks (We proposed this before - Draft of Testnet ideas - Google Docs). So we’re asking the question again: why does the testnet exist? The proposal mentions the following: “This program rewards validators for their contributions to maintaining and improving the Testnet.” where is the improving part? It seems it goes down the road of 'Let it run and pray it keeps running; by all means, don’t test anything because your uptime will be hit, and thus, your rewards will be hit.

  • The incentivisation (€50/100/200 per month) covers costs only (if that).

  • The proposed lottery system creates uncertainty about whether costs will be covered at all. Or are you expected to shut down and start up nodes every time? And shut down physical servers from quarter to quarter?

  • Are the 15 top-30-mainnet operators limited to participating in the €50/month incentivisation in your proposal? If so, why?

  • Why mention the optional incentives for mainnet validators here specifically? Everyone can spin up a testnet node anyway?

  • Partial rewards: How do you propose this works?

3. Validator Requirements:

  • How do you propose checking for minimum HW requirements?

4. Program Budget:

  • Lowering the budget by ~80% doesn’t inspire confidence and willingness to reward participants to use the testnet for its intended purpose. Can you explain why this amount was chosen and why it was lowered to 1/5th of the original amount?

Proposed changes:

  • Move away from a lottery system.
  • Implement a task system, rewarding actual testing and sharing findings.
  • Include CA delegations in the incentivisation; i.e., if you want a CA delegation, you need to run a TTL node. This could incentivise at least a couple of testnet nodes without spending money.
  • Increase the budget; you can’t lower the budget by 80% and expect a better testnet experience than before.
  • If the budget is limited, consider moving to a smaller incentivised testnet set. Do we NEED 100 validators?

Tl;dr
The rewards have been lowered significantly, which does not inspire confidence and willingness to reward participants. The actual incentivisation doesn’t change much from the current system and fails to reward actual testing (the intended purpose of the testnet). If we had on-chain governance, we would vote NO for this proposal in its current form.

4 Likes
5

Hello.

As testnet participants we want to know why budget has been strong cutted.

  1. Now all application process could fill? What with situation when testnet participants will be over 200-300? Only first 100 will recive a reward?

(500GB with RAID 1)

You cannot force one option, some people will want to have ZFS, RAID 5, 6, 10 etc. or in big server have separated vm/container (via proxmox for example) for node.

6

1. Lottery Selection for Validators:

Introduce a weighted lottery system where validators with prior experience, verified knowledge, and higher performance metrics have an increased chance of selection. This ensures that the Testnet benefits from experienced participants while still maintaining an element of fairness.

2. Community Engagement for Program Evaluation:

Introduce a community feedback mechanism that allows validators to voice their experiences, suggestions, and concerns at the end of each quarter. This feedback loop could be implemented through a regular survey or a dedicated discussion forum. The Casper Association can use this input to make data-driven adjustments to the program, ensuring it evolves in response to validator needs and network conditions.

3. Reward Distribution Transparency:

Implement an on-chain dashboard where validators can track how their rewards are accumulating. (replacement of google docs)

4. Improved Monitoring Tools:

Provide validators with automated monitoring tools integrated with the Casper Testnet’s network infrastructure. These tools could offer real-time alerts, performance analytics, and automated diagnostics. For example, automated notifications could inform validators of issues like high latency, port closures, or syncing problems. This proactive approach empowers validators to swiftly address problems, maintaining network stability. (Currenly validators only get to know if they’re experiencing any issues by era guardian at the end of each era. But problems usually occur during an era. The current calculations right now for the testnet can be improved and became a great tool for faster notifications - I can gladly help with this part. Would love to improve era guardian)

5. Educational Resources and Training:

Create a comprehensive educational program, including webinars, documentation, and an online certification process that validators must complete before applying. This training program ensures that all validators have a baseline understanding of Casper’s network, increasing overall network reliability. Additionally, validators who complete the training could receive a one-time bonus, incentivizing them to engage with the educational material. We can also use experienced testnet validators to do the trainings and reward them based on their work rather than random rewarding.

6. Incentive for Reporting Bugs and Issues:

Introduce a bug bounty program within the Testnet that rewards validators for reporting issues along with relevant logs. This approach encourages active participation in network maintenance and fosters a collaborative community effort to improve the Testnet. Validators who report critical bugs that lead to significant network improvements could receive bonus rewards, boosting engagement.

7. Top-30 Mainnet Validators:

  • Potential Pitfall: Requiring Top-30 validators to opt into one of 15 reserved slots might not fully capture their willingness to contribute without incentives. Also keeping track of top-30 validators is another challenge. It’s a dynamic list that can change a lot and create confusion.
  • Improvement: Allow Top-30 validators to participate in additional community roles beyond just validation. For example, they could act as mentors, provide network analytics, or contribute to documentation. By engaging Top-30 validators in various capacities, the program leverages their expertise while providing them with opportunities to influence the Testnet’s direction.

8. Feedback Loop for Disqualified Validators

  • Potential Pitfall: Disqualified validators may not understand the reasons for their disqualification, leading to frustration and reduced future participation.
  • Improvement: Implement a feedback system where disqualified validators receive a detailed report outlining the reasons for their disqualification, including metrics they failed to meet. This transparency helps validators learn from their mistakes and prepares them to reapply successfully in future quarters.
1 Like
7

For the new testnet program, I propose to leave the current rules in force with the following changes:

  • Make the scoring tool transparent and available online.
  • Introduce a separate quota for 10 full nodes that will receive increased rewards. Determine the operators eligible to launch such full nodes by means of a lottery in which participants who showed the best results in the previous period can participate.
  • Supplement the KYC verification procedure with the liveness check.
  • Set a deadline for processing the results and paying out rewards. For example, 14 days from the end of the period.
  • To make the use of funds more rational, reduce the size of the rewards in accordance with [Michael Steuer’s proposal.

Keeping the current rules, including scoring based on node performance and longevity, will allow rewarding and motivating those validators who are most responsible in maintaining their nodes and have been participating in the program for the longest time, while the lottery deprives participants of the motivation to pay enough attention to their nodes and will allow random / inexperienced participants to claim rewards.

8

While I agree with the suggestions of @Artem and @SpeedyStaking , I suggest an alternative model by stating that completely random selection does not make much sense. We can call it a weighted measurement model. In the previous model, many friends with “validate” status were victimized due to measurement error. Both for this reason and to be healthier, I thought of a model with a formula.

  • First, a set of rules will be created and the necessary weights will be defined.
  • Random_rate will be determined. My idea is 50%.

Since there will be no performance-based criteria in the weighting rules, all experimental operations can be performed at the desired level.

I designed a scenario as an example:

Based on last quarter’s data.

Here the rule entry is in the list “weights” , In order:

  • 15 % of the past test network ‘Longevity Score’,
  • If you have top30 nodes in the mainnet, it is 15 %,
  • 7.5 % if you are the mainnet validator,
  • 7.5 % if you know enough to manage a node,
  • Other rule (if required) is 5 %,

random_normalize_weights
random_normalize_weights980×371 16.5 KB

A set of rules with a total impact of 50%.

Last quarter testnet “longevity score” data has been normalized for inclusion. So it is compressed to 0-1.

As I don’t have the data from the form, the data here, except for the ‘Longevity Score’, has been randomly assigned to create a scenario.

random_features
random_features807×119 5.19 KB

The formula is obtained by multiplying each rule value by a weight value assigned to it. The maximum value is 1.0. The “random_rate” here is completely random. Its distribution is uniform. This means that the chances of the first and the last participant’s “random_rate” are the same.

fx

Here, a 4-quarter scenario was envisioned and it was %50 lottery and %50 rules based.

loop
loop986×325 22.4 KB

If we look at the results in general ,

The index is sorted by longevity score but the image is sorted by “selection_score” from low to high. According to this, your longevity score alone has no effect, but even if you do not have the other rules, if you can catch random_rate 0.5, you can also win a prize :slight_smile:

result_order_min
result_order_min1920×1015 107 KB

If we look at the availability of quarters in general,

reward_dist

In the last quarter there were a total of 218 nodes (according to the testnet scoreboard) and only 195 of them had a longevity_score different from 0.

As a result, the number of nodes that qualify to participate in any quarter in a year according to the designed scenario is as follows. Only 34 nodes participated in all 4 quarters and 59 nodes participated in at least 1 quarter. In total, 171 nodes could qualify for an award in 1 year.

Note that the “weight” values /rules and random_rate can be changed.

This idea can be improved and I was able to think of something similar in a short time. Just randomly choosing a node would largely ignore the people who have been managing the nodes for a long time. And it would also cause repeat node setups that are not guaranteed to happen quarter after quarter :slight_smile: With the right weighting ratios, something good could happen.

This lottery can be applied to both “ttl” and “fullnode” quotas.

Thanks for taking your time to read this long article :slight_smile:

9

I propose to allow applicants to specify up to 3 public keys in the application.
Thus, if the number of applications is insufficient (less than 100), it will be possible to hold an additional lottery among the same applicants to fill vacant slots.
As a result, some of the applicants will be able to launch two or three nodes. This will have the following positive effects:

  1. Reducing the labor costs of the organizers for administering the testnet (fewer participants and KYC) and simplifying communication.
  2. Increasing the motivation of participants due to an increase in the amount of potential rewards per participant, which is especially important in view of the sharp reduction in the budget.
  3. A smaller number of more motivated participants will be more responsive to possible unforeseen challenges requiring a quick and qualified response.
3 Likes
10

I like this proposal - ultimately it’s about getting people involved who want to be operating testnet nodes for the right reasons (e.g. to be better mainnet operators, to build dapps, to be ahead of mainnet with upgrades, etc.), not just to chase a profit on a cheap VPS.

@kara can you update the proposal to incorporate this idea?

1 Like
11

While I like this idea, I worry about using the old performance data to determine this, which would perpetuate the status quo. Can you propose a fair weighting mechanism that we can all evaluate?

I like this. @kara please add!

Not sure this is needed. If you are an active participant, and your performance is above 99%, then you’re rewards are accumulating at 100%?

I know there are some open source grafana board and other monitoring templates etc. made by the community. I know @sacherjj has some as well, that we may be able to make available. Or of course @EraGuardian could become the standard the testnet validators all use :slightly_smiling_face:

Good idea to aspire to, though not feasible in the near term where we really need to get this new program off the ground ASAP, since we’re already technically in the new era. @kara take note though please.

We could look at making a small budget available for that every quarter. We used to have that in place, and ever only got one bug bounty request through it.

I think we need to determine a way of determining the list, like a snap shot the week prior to each quarter beginning. @kara

Disqualification should only happen based on on-chain performance drops, fully transparent.

12

Good call. @kara please update

The management is handled by CA, not a cost to validators/participants

The testnet exists so that mainnet validators can test out new releases prior to them hitting mainnet, so that dapp developers can test their applications with “free token” and sometimes against upcoming versions of the protocol, and so that core developers can observe the behavior of new versions of the node/protocol in an environment more resemblant of mainnet, etc. To be clear, it is not a guaranteed income program.

it should cover cost and then some. The top validator in the network with consistent 100% performance runs on EUR 40/month machines. The EUR 50 option is just there for mainnet validators, who generally don’t want/need to be incentivized to be on testnet anyway. Everyone else gets between double to quadruple what the hard costs of running a node should be.

Correct, the certainty is provided 3 months at-the-time.

Only if they want to gain access to those 15 slots. But again, most mainnet validators are participating in testnet for other motivations than making a quick buck.

The point is to reserve some slots specifically for mainnet validators, because they have reasons to be there other than testnet incentives. Everyone can spin up a node, but not everyone is in the 100 validator node set. That’s determined by delegation.

On a pro-rate basis. @kara I assume you would get paid for the days that you were in compliance, and not paid for those that you were not? Please make this clearer in the text.

I don’t think we’d check that pro-actively for everyone. But if someone becomes disqualified and then objects, they’d have to proof they were running on minimum HW requirements, for example.

Budgetary constraints are what they are. On the flip side, with a lower token price you may get a lot of tokens still at a significantly lower tax basis!

Again, I think you misunderstand the use of the testnet. It’s not a job program. It is to test the protocol, by core developers, dapp developers and mainnet operators, in advance of it hitting mainnet.

13

Explained in the answers above

Yes, the incentivized portion of testnet will be limited to a set of 100 validators, to be determined as outlined in this (to-be-amended) proposal

No one is forcing anything, this was provided as a minimum example for a TTL node

1 Like
14

Thank you all for your great feedback on this proposal. Your insights and comments were very helpful and constructive. Considering we are now 13 days into October, I would like to wrap this up ASAP and get this program underway. @kara will take a few days to iterate the program based on the suggestions and improvements you all provided in this thread, and present a final version sometime later this week, which we can then proceed to implement.

Thanks again for all your participation in improving this proposal!

2 Likes
15

In case the latest proposal goes through 1 important note:
The automated bot should not be another blackbox like it was before to reduce friction and discussions later on.

There is also a confirmed “issue” that the SSE can fire a “BlockAdded” event but the RPC can’t return the latest block because it is not fully stored yet - I don’t know the exact time though. This can result in penalties which are out of the hand of a node operator since this is by design. Querying the RPC endpoint will result in an error.

That was confirmed some time ago by a core dev. Here is the message:

image
image476×134 5.6 KB

16

As I said in the Telegram group, and Michael agreed, I suggest we leave the ability to change the Program rules as we go along, by gathering the forum and voting again. Hopefully, even though it didn’t make it into the editorial, we will have that opportunity.

2 Likes
17

100%. Recently, the validators participated in a first on-chain vote about the future of the incentivized testnet program. Each validator’s voting power was allocated by the percentage of the network they represent, thereby representing the relative stake the community has entrusted them with. This process marks a first step on the roadmap to fuller on-chain decentralized governance. You can see the vote result here: Era Guardian

Anything in this proposal is subject to future revision by the validator community through subsequent votes.