- I made a mistake when specifying the `FOR` clause for the index hint,
I read it as being an required argument by XORM. The [MariaDB
documention](https://mariadb.com/kb/en/use-index/) tells that it
defaults to the `FOR JOIN` clause hence why I specified `JOIN` (As can
be seen in the previous PR's SQL analyze I didn't specify the `FOR`
clause). However apparently there seems to be some wizardy going on as
we need to tell MariaDB to use this index for the `ORDER BY` clause to
actually force MariaDB to use this index over the `updated_unix` index.
However because it's not actually required by XORM to specify this
value I leave this empty as mariadb is apparently smart enough to figure
out for which type we want to use this index.
- TL;DR make this index hint actually effective for MariaDB.
- Ref: #6146
- For the notifications page the unread and pinned notifications are
gathered for doer those that and are ordered by the updated unix.
MariaDB makes a bad decision (sometimes, for most users it does not make
this decision) with this query, it uses the index for the `updated_unix`
column to speed up this query, however this is not the correct index to
be taking, if the doer does not have more than 20 (the
page size) unread and pinned notifications combined MariaDB will
traverse the whole notifications table before it realizes that there are
no more notifications to be gathered. It instead should use the index
for the `user_id` column (this is what MariaDB already does for most
users), so the list that has to be traversed is limited to the doer's
notifications which is significantly less than the whole notifications
table.
- This is a different approach than what Gitea has taken to solve this
problem, which is to add a index to the (status, userid, updated_unix)
tuple (Ref: https://github.com/go-gitea/gitea/pull/32395). Adding more
and more indexes is not a good way if we can use existing indexes to get
a query to a acceptable performance.
- The code cannot use `db.Find` as it's hard to add a index hint option
specifically for this query and not for the other instances that uses
`activities_model.FindNotificationOptions`.
- Only add a index hint for MySQL as I have not been able to test if
SQLite or PostgreSQL are smart enough to use the better index (as you
need a large enough dataset to test this meaningfully).
- Integration test added to ensure the SQL is run by all databases.
---
Performance numbers (from Codeberg's database - MariaDB
10.11.6-MariaDB-0+deb12u1):
Currently:
```sql
SELECT * FROM `notification` WHERE notification.user_id=26734 AND (notification.status=3 OR notification.status=1) ORDER BY notification.updated_unix DESC LIMIT 20;
(5.731 sec)
+------+-------------+--------------+-------+--------------------------------------------------+-------------------------------+---------+-------+---------+------------+----------+------------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | r_rows | filtered | r_filtered | Extra |
+------+-------------+--------------+-------+--------------------------------------------------+-------------------------------+---------+-------+---------+------------+----------+------------+-------------+
| 1 | SIMPLE | notification | index | IDX_notification_status,IDX_notification_user_id | IDX_notification_updated_unix | 8 | const | 1376836 | 1474066.00 | 50.03 | 0.00 | Using where |
+------+-------------+--------------+-------+--------------------------------------------------+-------------------------------+---------+-------+---------+------------+----------+------------+-------------+
```
Using the better index:
```sql
SELECT * FROM `notification` USE INDEX (IDX_notification_user_id) WHERE notification.user_id=26734 AND (notification.status=3 OR notification.status=1) ORDER BY notification.updated_unix DESC LIMIT 20;
(0.834 sec)
+------+-------------+--------------+--------+----------------------------------------------------------+--------------------------+---------+----------------------------------+-------+----------+----------+------------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | r_rows | filtered | r_filtered | Extra |
+------+-------------+--------------+--------+----------------------------------------------------------+--------------------------+---------+----------------------------------+-------+----------+----------+------------+----------------------------------------------+
| 1 | PRIMARY | notification | ref | PRIMARY,IDX_notification_status,IDX_notification_user_id | IDX_notification_user_id | 8 | const | 22042 | 10756.00 | 50.03 | 0.02 | Using where; Using temporary; Using filesort |
| 1 | PRIMARY | notification | eq_ref | PRIMARY | PRIMARY | 8 | gitea_production.notification.id | 1 | 1.00 | 100.00 | 100.00 | |
+------+-------------+--------------+--------+----------------------------------------------------------+--------------------------+---------+----------------------------------+-------+----------+----------+------------+----------------------------------------------+
```
Fix#31807
ps: the newly added params's value will be changed.
When the first time you selected the filter, the values of params will
be `0` or `1`
But in pager it will be `true` or `false`.
So do we have `boolToInt` function?
(cherry picked from commit 7092402a2db255ecde2c20574b973fb632c16d2e)
Conflicts:
routers/web/org/home.go
trivial conflict s/pager.AddParam/pager.AddParamString/
Fix#26685
If a commit status comes from Gitea Actions and the user cannot access
the repo's actions unit (the user does not have the permission or the
actions unit is disabled), a 404 page will occur after clicking the
"Details" link. We should hide the "Details" link in this case.
<img
src="https://github.com/go-gitea/gitea/assets/15528715/68361714-b784-4bb5-baab-efde4221f466"
width="400px" />
(cherry picked from commit 7dec8de9147b20c014d68bb1020afe28a263b95a)
Conflicts:
routers/web/repo/commit.go
trivial context commit
This PR will avoid load pullrequest.Issue twice in pull request list
page. It will reduce x times database queries for those WIP pull
requests.
Partially fix#29585
---------
Co-authored-by: Giteabot <teabot@gitea.io>
(cherry picked from commit 62f8174aa2fae1481c7e17a6afcb731a5b178cd0)
Conflicts:
models/activities/notification_list.go
moved to models/activities/notification.go
just some refactoring bits towards replacing **util.OptionalBool** with
**optional.Option[bool]**
---------
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
(cherry picked from commit f6656181e4a07d6c415927220efa2077d509f7c6)
Conflicts:
models/repo/repo_list_test.go
trivial shared fixture count conflicts
Since `modules/context` has to depend on `models` and many other
packages, it should be moved from `modules/context` to
`services/context` according to design principles. There is no logic
code change on this PR, only move packages.
- Move `code.gitea.io/gitea/modules/context` to
`code.gitea.io/gitea/services/context`
- Move `code.gitea.io/gitea/modules/contexttest` to
`code.gitea.io/gitea/services/contexttest` because of depending on
context
- Move `code.gitea.io/gitea/modules/upload` to
`code.gitea.io/gitea/services/context/upload` because of depending on
context
(cherry picked from commit 29f149bd9f517225a3c9f1ca3fb0a7b5325af696)
Conflicts:
routers/api/packages/alpine/alpine.go
routers/api/v1/repo/issue_reaction.go
routers/install/install.go
routers/web/admin/config.go
routers/web/passkey.go
routers/web/repo/search.go
routers/web/repo/setting/default_branch.go
routers/web/user/home.go
routers/web/user/profile.go
tests/integration/editor_test.go
tests/integration/integration_test.go
tests/integration/mirror_push_test.go
trivial context conflicts
also modified all other occurrences in Forgejo specific files
Fix#24662.
Replace #24822 and #25708 (although it has been merged)
## Background
In the past, Gitea supported issue searching with a keyword and
conditions in a less efficient way. It worked by searching for issues
with the keyword and obtaining limited IDs (as it is heavy to get all)
on the indexer (bleve/elasticsearch/meilisearch), and then querying with
conditions on the database to find a subset of the found IDs. This is
why the results could be incomplete.
To solve this issue, we need to store all fields that could be used as
conditions in the indexer and support both keyword and additional
conditions when searching with the indexer.
## Major changes
- Redefine `IndexerData` to include all fields that could be used as
filter conditions.
- Refactor `Search(ctx context.Context, kw string, repoIDs []int64,
limit, start int, state string)` to `Search(ctx context.Context, options
*SearchOptions)`, so it supports more conditions now.
- Change the data type stored in `issueIndexerQueue`. Use
`IndexerMetadata` instead of `IndexerData` in case the data has been
updated while it is in the queue. This also reduces the storage size of
the queue.
- Enhance searching with Bleve/Elasticsearch/Meilisearch, make them
fully support `SearchOptions`. Also, update the data versions.
- Keep most logic of database indexer, but remove
`issues.SearchIssueIDsByKeyword` in `models` to avoid confusion where is
the entry point to search issues.
- Start a Meilisearch instance to test it in unit tests.
- Add unit tests with almost full coverage to test
Bleve/Elasticsearch/Meilisearch indexer.
---------
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Fix#25627
1. `ctx.Data["Link"]` should use relative URL but not AppURL
2. The `data-params` is incorrect because it doesn't contain "page". JS
can simply use "window.location.search" to construct the AJAX URL
3. The `data-xxx` and `id` in notification_subscriptions.tmpl were
copied&pasted, they don't have affect.
The `q` parameter was not rendered in pagination links because
`context.Pagination:AddParam` checks for existance of the parameter in
`ctx.Data` where it was absent. Added the parameter there to fix it.
- Move the file `compare.go` and `slice.go` to `slice.go`.
- Fix `ExistsInSlice`, it's buggy
- It uses `sort.Search`, so it assumes that the input slice is sorted.
- It passes `func(i int) bool { return slice[i] == target })` to
`sort.Search`, that's incorrect, check the doc of `sort.Search`.
- Conbine `IsInt64InSlice(int64, []int64)` and `ExistsInSlice(string,
[]string)` to `SliceContains[T]([]T, T)`.
- Conbine `IsSliceInt64Eq([]int64, []int64)` and `IsEqualSlice([]string,
[]string)` to `SliceSortedEqual[T]([]T, T)`.
- Add `SliceEqual[T]([]T, T)` as a distinction from
`SliceSortedEqual[T]([]T, T)`.
- Redesign `RemoveIDFromList([]int64, int64) ([]int64, bool)` to
`SliceRemoveAll[T]([]T, T) []T`.
- Add `SliceContainsFunc[T]([]T, func(T) bool)` and
`SliceRemoveAllFunc[T]([]T, func(T) bool)` for general use.
- Add comments to explain why not `golang.org/x/exp/slices`.
- Add unit tests.
Change all license headers to comply with REUSE specification.
Fix#16132
Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
This PR adds a context parameter to a bunch of methods. Some helper
`xxxCtx()` methods got replaced with the normal name now.
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Adds GitHub-like pages to view watched repos and subscribed issues/PRs
This is my second try to fix this, but it is better than the first since
it doesn't uses a filter option which could be slow when accessing
`/issues` or `/pulls` and it shows both pulls and issues (the first try
is #17053).
Closes#16111
Replaces and closes#17053
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
If the context is cancelled `.NotificationUnreadCount` in a template can
cause an infinite loop with `ctx.ServerError()` being called, which
creates a template that then calls `.NotificationUnreadCount` calling
`GetNotificationCount()` with the cancelled context resulting in an
error that calls `ctx.ServerError`... and so on...
This PR simply stops calling `ctx.ServerError` in the error handler code
for `.NotificationUnreadCount` as we have already started rendering and
so it is too late to call `ctx.ServerError`. Additionally we skip
logging the error if it's a context cancelled error.
Fix#19793
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Reusing `/api/v1` from Gitea UI Pages have pros and cons.
Pros:
1) Less code copy
Cons:
1) API/v1 have to support shared session with page requests.
2) You need to consider for each other when you want to change something about api/v1 or page.
This PR moves all dependencies to API/v1 from UI Pages.
Partially replace #16052
There are multiple places where Gitea does not properly escape URLs that it is building and there are multiple places where it builds urls when there is already a simpler function available to use this.
This is an extensive PR attempting to fix these issues.
1. The first commit in this PR looks through all href, src and links in the Gitea codebase and has attempted to catch all the places where there is potentially incomplete escaping.
2. Whilst doing this we will prefer to use functions that create URLs over recreating them by hand.
3. All uses of strings should be directly escaped - even if they are not currently expected to contain escaping characters. The main benefit to doing this will be that we can consider relaxing the constraints on user names and reponames in future.
4. The next commit looks at escaping in the wiki and re-considers the urls that are used there. Using the improved escaping here wiki files containing '/'. (This implementation will currently still place all of the wiki files the root directory of the repo but this would not be difficult to change.)
5. The title generation in feeds is now properly escaped.
6. EscapePound is no longer needed - urls should be PathEscaped / QueryEscaped as necessary but then re-escaped with Escape when creating html with locales Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Apache `ProxyPassReverse` only works for Location, Content-Location and URI headers on HTTP redirect responses, it causes more problems than it resolves. Now all URLs generated by Gitee have the correct prefix AppSubURL. We do not need to set `ProxyPassReverse`.
* fix url param
* use AppSubURL instead of AppURL in api/v1
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Followup from #16562 prepare for #16567
* Rename ctx.Form() to ctx.FormString()
* Reimplement FormX func to need less code and cpu cycles
* Move code into own file
When marking notifications read the results may be returned out of order
or be delayed. This PR sends a sequence number to gitea so that the
browser can ensure that only the results of the latest notification
change are shown.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
* refactor routers directory
* move func used for web and api to common
* make corsHandler a function to prohibit side efects
* rm unused func
Co-authored-by: 6543 <6543@obermui.de>
2021-06-09 01:33:54 +02:00
Renamed from routers/user/notification.go (Browse further)
