Skip to content
This repository was archived by the owner on Aug 13, 2020. It is now read-only.

Commit 91d42b8

Browse files
hopsofthopsoft
authored
Change to a single impressions query for daily summary rollup (#1119)
* Change to a single impressions query for daily summary rollup * Get seeds working again and running much faster * Update daily summary job to use single stat query * Update local cache size to be configurable * Update and run standard * Patching up the test suite * Getting closer on patching up the test suite * Progress on the test suite * More work on test suite * More test contortions * More work on test suite and setup * Fix system tests * Add rake task for importing impressions
1 parent 7d9ea9b commit 91d42b8

File tree

109 files changed

+962
-1464
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

109 files changed

+962
-1464
lines changed

‎Gemfile

+1-3
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,6 @@ gem "acts_as_commentable_with_threading", "~> 2.0.1"
1111
gem "aws-sdk-s3", "~> 1.48.0", require: false
1212
gem "barnes", "~> 0.0.7"
1313
gem "bootsnap", ">= 1.4.2", require: false
14-
gem "bulk_insert", "~> 1.7.0"
1514
gem "cable_ready", "~> 4.0.7"
1615
gem "camo", "~> 0.1.0"
1716
gem "chroma", "~> 0.2.0"
@@ -100,7 +99,6 @@ group :development do
10099
gem "annotate"
101100
gem "bullet"
102101
gem "erb_lint"
103-
gem "factory_bot_rails"
104102
gem "letter_opener_web"
105103
gem "listen", "~> 3.2"
106104
gem "mechanize"
@@ -109,7 +107,7 @@ group :development do
109107
gem "ngrok-tunnel"
110108
gem "rack-mini-profiler", require: false
111109
gem "solargraph"
112-
gem "standard", ">= 0.1"
110+
gem "standard"
113111
gem "teamocil"
114112
gem "tocer"
115113
gem "tty-box"

‎Gemfile.lock

+3-12
Original file line numberDiff line numberDiff line change
@@ -114,8 +114,6 @@ GEM
114114
bootsnap (1.4.6)
115115
msgpack (~> 1.0)
116116
builder (3.2.4)
117-
bulk_insert (1.7.0)
118-
activerecord (>= 3.2.0)
119117
bullet (6.1.0)
120118
activesupport (>= 3.0.0)
121119
uniform_notifier (~> 1.11)
@@ -199,11 +197,6 @@ GEM
199197
ethon (0.12.0)
200198
ffi (>= 1.3.0)
201199
execjs (2.7.0)
202-
factory_bot (5.1.1)
203-
activesupport (>= 4.2.0)
204-
factory_bot_rails (5.1.1)
205-
factory_bot (~> 5.1.0)
206-
railties (>= 4.2.0)
207200
faker (2.10.2)
208201
i18n (>= 1.6, < 2)
209202
ferrum (0.7)
@@ -328,7 +321,7 @@ GEM
328321
activerecord (>= 4.2)
329322
request_store (~> 1.1)
330323
parallel (1.19.1)
331-
parser (2.7.0.2)
324+
parser (2.7.0.4)
332325
ast (~> 2.4.0)
333326
pastel (0.7.3)
334327
equatable (~> 0.6)
@@ -512,7 +505,7 @@ GEM
512505
actionpack (>= 4.0)
513506
activesupport (>= 4.0)
514507
sprockets (>= 3.0.0)
515-
standard (0.1.10)
508+
standard (0.2.0)
516509
rubocop (~> 0.79.0)
517510
rubocop-performance (~> 1.5.1)
518511
statsd-ruby (1.4.0)
@@ -600,7 +593,6 @@ DEPENDENCIES
600593
aws-sdk-s3 (~> 1.48.0)
601594
barnes (~> 0.0.7)
602595
bootsnap (>= 1.4.2)
603-
bulk_insert (~> 1.7.0)
604596
bullet
605597
byebug
606598
cable_ready (~> 4.0.7)
@@ -622,7 +614,6 @@ DEPENDENCIES
622614
dotenv-rails
623615
email_address (~> 0.1.11)
624616
erb_lint
625-
factory_bot_rails
626617
faker
627618
full-name-splitter (~> 0.1.2)
628619
hashdiff
@@ -687,7 +678,7 @@ DEPENDENCIES
687678
solargraph
688679
split (~> 3.4.1)
689680
spreadsheet (~> 1.2.4)
690-
standard (>= 0.1)
681+
standard
691682
stimulus_reflex (~> 2.1.9)
692683
stopwords-filter (~> 0.4.1)
693684
stripe (~> 5.4.1)

‎Procfile

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
11
web: bin/start-pgbouncer-stunnel bin/start-nginx bundle exec puma -C config/puma.rb
22
worker: bin/start-pgbouncer-stunnel bundle exec sidekiq -C config/sidekiq.yml
3-
worker_daily_summaries: bin/start-pgbouncer-stunnel bundle exec sidekiq -C config/sidekiq_daily_summaries.yml
3+
worker_data: bin/start-pgbouncer-stunnel bundle exec sidekiq -C config/sidekiq_data.yml
44
release: bundle exec rails db:migrate

‎app/channels/application_cable/connection.rb

+1-1
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ def connect
1414
self.ids = {
1515
current_user_id: current_user_id,
1616
true_user_id: true_user_id,
17-
session_id: session_id,
17+
session_id: session_id
1818
}
1919

2020
Rails.logger.debug "CONNECT: #{ids.inspect}"

‎app/controllers/advertisement_clicks_controller.rb

+1-1
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@ def mustache_params
5151
property_name: CGI.escape(@property&.name.to_s),
5252
property_url: CGI.escape(@property&.url.to_s),
5353
template: CGI.escape(@template.to_s),
54-
theme: CGI.escape(@theme.to_s),
54+
theme: CGI.escape(@theme.to_s)
5555
}
5656
end
5757

‎app/controllers/advertisements_controller.rb

+3-3
Original file line numberDiff line numberDiff line change
@@ -106,7 +106,7 @@ def set_advertisement_variables
106106
creative_id: @creative.id,
107107
property_id: property.id,
108108
template: template_name,
109-
theme: theme_name,
109+
theme: theme_name
110110
)
111111
@impression_url = impression_url(@virtual_impression_id, format: :gif)
112112
@powered_by_url = referral_code ? invite_url(referral_code) : root_url
@@ -343,7 +343,7 @@ def track_event(event_name)
343343
creative: @creative,
344344
ad_template: template_name,
345345
ad_theme: theme_name,
346-
country_code: country_code,
346+
country_code: country_code
347347
).track_event(event_name)
348348
end
349349

@@ -357,7 +357,7 @@ def create_virtual_impression
357357
creative_id: @creative.id,
358358
ad_template: template_name,
359359
ad_theme: theme_name,
360-
ip_address: ip_address,
360+
ip_address: ip_address
361361
}, expires_in: 30.seconds
362362

363363
track_event :virtual_impression_created

‎app/controllers/application_controller.rb

+5-5
Original file line numberDiff line numberDiff line change
@@ -93,11 +93,11 @@ def set_meta_tag_data
9393
{href: "/apple-touch-icon.png", sizes: "180x180", rel: "apple-touch-icon", type: "image/png"},
9494
{href: "/favicon-32x32.png", sizes: "32x32 96x96", rel: "icon", type: "image/png"},
9595
{href: "/favicon-16x16.png", sizes: "16x16", rel: "icon", type: "image/png"},
96-
{href: "/safari-pinned-tab.svg", rel: "mask-icon", color: "#5bbad5", type: "image/svg"},
96+
{href: "/safari-pinned-tab.svg", rel: "mask-icon", color: "#5bbad5", type: "image/svg"}
9797
],
9898
manifest: "/site.webmanifest",
9999
alternate: [
100-
{href: "#{ENV["WORDPRESS_URL"]}/blog/feed", type: "application/rss+xml", title: "RSS"},
100+
{href: "#{ENV["WORDPRESS_URL"]}/blog/feed", type: "application/rss+xml", title: "RSS"}
101101
],
102102
"apple-mobile-web-app-title": "CodeFund",
103103
"application-name": "CodeFund",
@@ -111,12 +111,12 @@ def set_meta_tag_data
111111
site_name: "CodeFund",
112112
locale: "en_US",
113113
fb: {
114-
admins: "801507563",
115-
},
114+
admins: "801507563"
115+
}
116116
},
117117
twitter: {
118118
card: "photo",
119-
site: "@codefundio",
119+
site: "@codefundio"
120120
},
121121
viewport: "width=device-width, initial-scale=1, shrink-to-fit=no"
122122
)

‎app/controllers/campaign_dashboards_controller.rb

+2-2
Original file line numberDiff line numberDiff line change
@@ -10,8 +10,8 @@ def show
1010
params: {
1111
"campaign_id" => @campaign.id,
1212
"start_date" => @start_date.strftime("%F"),
13-
"end_date" => @end_date.strftime("%F"),
14-
},
13+
"end_date" => @end_date.strftime("%F")
14+
}
1515
}
1616
token = JWT.encode payload, ENV["METABASE_SECRET_KEY"]
1717

‎app/controllers/campaign_reports_controller.rb

+1-1
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ def campaign_mailer_params
1919
to: campaign_report_params[:email],
2020
campaign: @campaign,
2121
start_date: @start_date.iso8601,
22-
end_date: @end_date.iso8601,
22+
end_date: @end_date.iso8601
2323
}
2424
end
2525

‎app/controllers/campaigns_controller.rb

+3-3
Original file line numberDiff line numberDiff line change
@@ -20,13 +20,13 @@ def show
2020

2121
payload = {
2222
resource: {
23-
dashboard: ENV["METABASE_CAMPAIGN_DASHBOARD_ID"].to_i,
23+
dashboard: ENV["METABASE_CAMPAIGN_DASHBOARD_ID"].to_i
2424
},
2525
params: {
2626
"campaign_id" => @campaign.id,
2727
"start_date" => @start_date.strftime("%F"),
28-
"end_date" => @end_date.strftime("%F"),
29-
},
28+
"end_date" => @end_date.strftime("%F")
29+
}
3030
}
3131
token = JWT.encode payload, ENV["METABASE_SECRET_KEY"]
3232

‎app/controllers/concerns/campaigns/stashable.rb

+1-1
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,7 @@ def stashed_campaign_params
4848
temporary_id: Campaign.maximum(:id) + 1,
4949
status: ENUMS::CAMPAIGN_STATUSES::PENDING,
5050
start_date: Date.tomorrow,
51-
ecpm: Money.new(ENV.fetch("BASE_ECPM", 400).to_i, "USD"),
51+
ecpm: Money.new(ENV.fetch("BASE_ECPM", 400).to_i, "USD")
5252
}
5353
)
5454
end

‎app/controllers/concerns/properties/stashable.rb

+1-1
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ def stashed_property_params
2929
temporary_id: Property.count + 1,
3030
status: ENUMS::PROPERTY_STATUSES::PENDING,
3131
ad_template: ENUMS::AD_TEMPLATES::DEFAULT,
32-
ad_theme: ENUMS::AD_THEMES::LIGHT,
32+
ad_theme: ENUMS::AD_THEMES::LIGHT
3333
}
3434
)
3535
end

‎app/controllers/global_stats_controller.rb

+1-1
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ def show
99
render json: {impressions_count: imp_count,
1010
properties_count: prop_count,
1111
campaigns_count: camp_count,
12-
click_rate: click_rate,},
12+
click_rate: click_rate},
1313
status: :ok
1414
end
1515

‎app/controllers/job_postings_controller.rb

+1-1
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ def show
4444
@similar_job_postings = job_postings.limit(12)
4545
@report_job_post_link = "mailto:[email protected]?#{{
4646
subject: "[Report Job] #{@job_posting.title}",
47-
body: "Link: #{job_posting_url(@job_posting)}\n\nI am reporting this job because ...\n\n",
47+
body: "Link: #{job_posting_url(@job_posting)}\n\nI am reporting this job because ...\n\n"
4848
}.to_query}".gsub("+", "%20")
4949
IncrementJobPostingViewsJob.perform_later @job_posting.id, "detail_view_count" unless device.bot?
5050
end

‎app/controllers/omniauth_callbacks_controller.rb

+1-1
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ def github
88
company_name: raw_info[:company],
99
bio: raw_info[:bio],
1010
website_url: raw_info[:blog],
11-
github_username: raw_info[:login],
11+
github_username: raw_info[:login]
1212
})
1313
end
1414

‎app/controllers/organizations_controller.rb

+3-3
Original file line numberDiff line numberDiff line change
@@ -13,13 +13,13 @@ def index
1313
def show
1414
payload = {
1515
resource: {
16-
dashboard: ENV["METABASE_ADVERTISER_DASHBOARD_ID"].to_i,
16+
dashboard: ENV["METABASE_ADVERTISER_DASHBOARD_ID"].to_i
1717
},
1818
params: {
1919
"organization_id" => @organization.id,
2020
"start_date" => @start_date.strftime("%F"),
21-
"end_date" => @end_date.strftime("%F"),
22-
},
21+
"end_date" => @end_date.strftime("%F")
22+
}
2323
}
2424
token = JWT.encode payload, ENV["METABASE_SECRET_KEY"]
2525

‎app/controllers/pricings_controller.rb

+1-1
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ def show
1515
name: country.name,
1616
value: country.ecpm(base: @base, multiplier: @multiplier).to_f,
1717
display_price: country.ecpm(base: @base, multiplier: @multiplier).format,
18-
emoji_flag: country.emoji_flag,
18+
emoji_flag: country.emoji_flag
1919
}
2020
@regions[country.region][country.subregion][:countries] << c
2121
@countries << c

‎app/controllers/properties_controller.rb

+2-2
Original file line numberDiff line numberDiff line change
@@ -40,8 +40,8 @@ def show
4040
params: {
4141
"property_id" => @property.id,
4242
"start_date" => @start_date.strftime("%F"),
43-
"end_date" => @end_date.strftime("%F"),
44-
},
43+
"end_date" => @end_date.strftime("%F")
44+
}
4545
}
4646
token = JWT.encode payload, ENV["METABASE_SECRET_KEY"]
4747

‎app/controllers/property_dashboards_controller.rb

+2-2
Original file line numberDiff line numberDiff line change
@@ -8,8 +8,8 @@ def show
88
params: {
99
"property_id" => @property.id,
1010
"start_date" => @start_date.strftime("%F"),
11-
"end_date" => @end_date.strftime("%F"),
12-
},
11+
"end_date" => @end_date.strftime("%F")
12+
}
1313
}
1414
token = JWT.encode payload, ENV["METABASE_SECRET_KEY"]
1515

‎app/controllers/search_controller.rb

+1-1
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ def show
99
users: User.search_name(q).or(User.search_email(q)).order(created_at: :desc).limit(5),
1010
properties: Property.search_name(q).order(created_at: :desc).limit(5),
1111
campaigns: Campaign.search_name(q).order(created_at: :desc).limit(5),
12-
organizations: Organization.search_name(q).order(created_at: :desc).limit(5),
12+
organizations: Organization.search_name(q).order(created_at: :desc).limit(5)
1313
}
1414
render layout: false
1515
end

‎app/controllers/wordpress_snippets_controller.rb

+1-1
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ def render_pricing_table
3030
name: country.name,
3131
value: country.ecpm(base: @base, multiplier: @multiplier).to_f,
3232
display_price: country.ecpm(base: @base, multiplier: @multiplier).format,
33-
emoji_flag: country.emoji_flag,
33+
emoji_flag: country.emoji_flag
3434
}
3535
@regions[country.region][country.subregion][:countries] << c
3636
@countries << c

‎app/helpers/advertisements_helper.rb

+4-4
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ def advertisement_mustache_template_options
1919
campaign: @campaign_url.to_s.strip,
2020
poweredBy: @powered_by_url.to_s.strip,
2121
adblock: ENV["ADBLOCK_PLUS_PIXEL_URL"].to_s.strip,
22-
uplift: @uplift_url.to_s.strip,
22+
uplift: @uplift_url.to_s.strip
2323
},
2424
creative: {
2525
name: @creative&.name,
@@ -30,9 +30,9 @@ def advertisement_mustache_template_options
3030
icon: @creative&.icon_image&.cloudfront_url,
3131
small: @creative&.small_image&.cloudfront_url,
3232
large: @creative&.large_image&.cloudfront_url,
33-
wide: @creative&.wide_image&.cloudfront_url,
34-
},
35-
},
33+
wide: @creative&.wide_image&.cloudfront_url
34+
}
35+
}
3636
}
3737
end
3838
end

‎app/helpers/application_helper.rb

+1-1
Original file line numberDiff line numberDiff line change
@@ -146,7 +146,7 @@ def noty_flash
146146
type = "error" if type == "alert"
147147
body = {
148148
type: type,
149-
text: message,
149+
text: message
150150
}
151151
text = "<script>new Noty(#{body.to_json}).show();</script>"
152152
flash_messages << text.html_safe if message

‎app/helpers/job_posting_purchases_helper.rb

+1-1
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ module JobPostingPurchasesHelper
22
def read_the_docs_keywords
33
[
44
"Machine Learning",
5-
"Python",
5+
"Python"
66
]
77
end
88
end

‎app/helpers/properties_helper.rb

+1-1
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ def property_responsive_behaviors_for_select
1515
[
1616
["Show ad using the configured template and theme", ENUMS::PROPERTY_RESPONSIVE_BEHAVIORS::NONE],
1717
["Show ad using a sticky footer that can be closed by the user", ENUMS::PROPERTY_RESPONSIVE_BEHAVIORS::FOOTER],
18-
["Do not show ads", ENUMS::PROPERTY_RESPONSIVE_BEHAVIORS::HIDE],
18+
["Do not show ads", ENUMS::PROPERTY_RESPONSIVE_BEHAVIORS::HIDE]
1919
]
2020
end
2121

‎app/jobs/create_daily_summary_job.rb

+12-13
Original file line numberDiff line numberDiff line change
@@ -8,19 +8,18 @@ def perform(impressionable, date_string, scoped_by, scoped_by_type = nil)
88
return if impressionable.is_a?(Campaign) && !impressionable.available_on?(date)
99
return unless impressionable.impressions.on(date).scoped_by(scoped_by, scoped_by_type).exists?
1010

11-
impressions = impressionable.impressions.on(date).scoped_by(scoped_by, scoped_by_type)
12-
13-
DailySummary.transaction do
14-
impressionable.daily_summaries.on(date).scoped_by(scoped_by, scoped_by_type).first_or_create!(
15-
impressions_count: impressions.count,
16-
fallbacks_count: impressions.fallback.count,
17-
clicks_count: impressions.clicked.count,
18-
unique_ip_addresses_count: impressions.distinct.count(:ip_address),
19-
gross_revenue: Money.new(impressions.sum(:estimated_gross_revenue_fractional_cents).round, "USD"),
20-
property_revenue: Money.new(impressions.sum(:estimated_property_revenue_fractional_cents).round, "USD"),
21-
house_revenue: Money.new(impressions.sum(:estimated_house_revenue_fractional_cents).round, "USD"),
22-
)
23-
end
11+
rollup = Impression.connection.exec_query(
12+
impressionable.impressions.on(date).scoped_by(scoped_by, scoped_by_type)
13+
.select(Arel.star.count.as("impressions_count"))
14+
.select("count(*) FILTER (WHERE fallback_campaign = true) AS fallbacks_count")
15+
.select("count(*) FILTER (WHERE clicked_at_date IS NOT NULL) AS clicks_count")
16+
.select("count(DISTINCT ip_address) AS unique_ip_addresses_count")
17+
.select(Arel::Nodes::NamedFunction.new("ROUND", [Impression.arel_table[:estimated_gross_revenue_fractional_cents].sum]).as("gross_revenue"))
18+
.select(Arel::Nodes::NamedFunction.new("ROUND", [Impression.arel_table[:estimated_property_revenue_fractional_cents].sum]).as("property_revenue"))
19+
.select(Arel::Nodes::NamedFunction.new("ROUND", [Impression.arel_table[:estimated_house_revenue_fractional_cents].sum]).as("house_revenue"))
20+
.to_sql
21+
).first
22+
impressionable.daily_summaries.on(date).scoped_by(scoped_by, scoped_by_type).first_or_create!(rollup)
2423
rescue ActiveRecord::RecordNotUnique => e
2524
logger.info "A race condition attempted to create the same DailySummary record more than once. #{e}"
2625
end

0 commit comments

Comments
 (0)