fix: корректные source_url для VK постов (screen_name формат)

- VkSource: добавлено поле screen_name, заполняется из VK API при добавлении источника - vk_parser: source_url теперь в формате vk.com/{screen_name}?w=wall{owner}_{post} вместо устаревшего vk.com/wall{owner}_{post} - _vk_post_exists: проверка через LIKE чтобы находить оба формата URL - admin_vk: автоматически получает screen_name при создании/обновлении источника - admin_system: исправлен regex в repair-media-urls — только /news-media/ URL - Alembic миграция: b2c3d4e5f6a7 добавляет колонку screen_name в vk_sources Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
fix: исправить URL медиа — nginx /media/ проксирует без имени бакета
2026-05-20 19:32:28 +05:00 · 2026-05-20 16:44:36 +05:00 · 2026-05-19 19:08:13 +05:00 · 2026-05-19 18:39:18 +05:00 · 2026-05-19 17:53:55 +05:00
10 changed files with 170 additions and 19 deletions
--- a/.env.example
+++ b/.env.example
@@ -1,12 +1,14 @@
 # ============= БАЗА ДАННЫХ =============
-DATABASE_URL=postgresql+asyncpg://news_admin:news_secure_2026@192.168.1.11:5432/news
+DATABASE_URL=postgresql+asyncpg://db_user:db_password@db-host:5432/news

 # ============= MINIO / ХРАНИЛИЩЕ =============
-MINIO_ENDPOINT=minio:9000
-MINIO_ACCESS_KEY=minioadmin
-MINIO_SECRET_KEY=minioadmin
+MINIO_ENDPOINT=minio-host:9000
+MINIO_ACCESS_KEY=your-access-key
+MINIO_SECRET_KEY=your-secret-key
 MINIO_BUCKET=news-media
-MINIO_PUBLIC_URL=http://localhost:9000
+# MINIO_PUBLIC_URL — публичный URL через который браузер загружает медиа.
+# nginx проксирует /media/ → MinIO bucket, поэтому ставим домен сайта + /media (без имени бакета)
+MINIO_PUBLIC_URL=https://your-domain.ru/media

 # ============= JWT / БЕЗОПАСНОСТЬ =============
 JWT_SECRET=your-super-secret-key-change-in-production
--- a/api/bd/models.py
+++ b/api/bd/models.py
@@ -82,6 +82,7 @@ class VkSource(Base):
    id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
    group_id: Mapped[str] = mapped_column(String(50), unique=True)   # числовой ID без минуса
    group_name: Mapped[str] = mapped_column(String(200))              # станет названием категории
+    screen_name: Mapped[str | None] = mapped_column(String(200), nullable=True)  # screen_name для корректных URL
    enabled: Mapped[bool] = mapped_column(Boolean, default=True)
    strict_filter: Mapped[bool] = mapped_column(Boolean, default=True)  # True = только посты с тегами
    last_run: Mapped[datetime | None] = mapped_column(DateTime, nullable=True)
--- a/api/migrations/versions/b2c3d4e5f6a7_add_screen_name_to_vk_sources.py
+++ b/api/migrations/versions/b2c3d4e5f6a7_add_screen_name_to_vk_sources.py
@@ -0,0 +1,26 @@
+"""add screen_name to vk_sources
+
+Revision ID: b2c3d4e5f6a7
+Revises: a1b2c3d4e5f6
+Create Date: 2026-05-20 18:00:00.000000
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+revision: str = 'b2c3d4e5f6a7'
+down_revision: Union[str, None] = 'a1b2c3d4e5f6'
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.add_column('vk_sources',
+        sa.Column('screen_name', sa.String(200), nullable=True)
+    )
+
+
+def downgrade() -> None:
+    op.drop_column('vk_sources', 'screen_name')
--- a/api/route/admin_system.py
+++ b/api/route/admin_system.py
@@ -113,3 +113,61 @@ async def system_status(db: AsyncSession = Depends(get_db), _: str = Depends(req
        "redis":    redis,
        "minio":    minio,
    }
+
+
+@router.post("/repair-media-urls")
+async def repair_media_urls(db: AsyncSession = Depends(get_db), _: str = Depends(require_admin)):
+    """Заменяет старые MinIO URL на актуальный MINIO_PUBLIC_URL во всех статьях и медиафайлах.
+
+    Системный nginx проксирует /media/ → MinIO bucket /news-media/, поэтому bucket исключается из URL.
+    Старый формат: http://host:port/news-media/object_path
+    Новый формат:  MINIO_PUBLIC_URL/object_path  (без bucket)
+
+    Запускать один раз после изменения MINIO_PUBLIC_URL в .env.
+    """
+    new_base = MINIO_PUBLIC_URL.rstrip("/")
+
+    # cover_url: http://host/news-media/path → new_base/path
+    r_cover = await db.execute(text(r"""
+        UPDATE articles
+        SET cover_url = :new_base || '/' ||
+            regexp_replace(cover_url, '^https?://[^/]+(:[0-9]+)?/[^/]+/', '')
+        WHERE cover_url IS NOT NULL
+          AND cover_url LIKE '%/news-media/%'
+          AND cover_url NOT LIKE :prefix
+        RETURNING id
+    """), {"new_base": new_base, "prefix": new_base + "%"})
+
+    # HTML-контент: только MinIO URL вида src="http://host/news-media/path"
+    # Точный паттерн: URL содержит /news-media/ — не трогаем другие ссылки
+    r_content = await db.execute(text(r"""
+        UPDATE articles
+        SET content = regexp_replace(
+            content,
+            'https?://[^/''"]+(:[0-9]+)?/news-media/([^''">]+)',
+            :new_base || '/\2',
+            'g'
+        )
+        WHERE content LIKE '%/news-media/%'
+          AND content NOT LIKE :prefix
+        RETURNING id
+    """), {"new_base": new_base, "prefix": "%" + new_base + "%"})
+
+    # media.url
+    r_media = await db.execute(text(r"""
+        UPDATE media
+        SET url = :new_base || '/' ||
+            regexp_replace(url, '^https?://[^/]+(:[0-9]+)?/[^/]+/', '')
+        WHERE url LIKE '%/news-media/%'
+          AND url NOT LIKE :prefix
+        RETURNING id
+    """), {"new_base": new_base, "prefix": new_base + "%"})
+
+    await db.commit()
+
+    return {
+        "new_base":        new_base,
+        "cover_url_fixed": r_cover.rowcount,
+        "content_fixed":   r_content.rowcount,
+        "media_fixed":     r_media.rowcount,
+    }
--- a/api/route/admin_vk.py
+++ b/api/route/admin_vk.py
@@ -1,3 +1,4 @@
+import httpx
 from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException  # BackgroundTasks используется в history endpoints
 from pydantic import BaseModel
 from sqlalchemy import select
@@ -18,11 +19,30 @@ class SourceIn(BaseModel):
    strict_filter: bool = True


+async def _fetch_screen_name(group_id: str) -> str | None:
+    if not VK_TOKEN:
+        return None
+    try:
+        async with httpx.AsyncClient(timeout=5) as client:
+            r = await client.get(
+                "https://api.vk.com/method/groups.getById",
+                params={"group_ids": group_id, "access_token": VK_TOKEN, "v": "5.131"},
+            )
+            data = r.json()
+            groups = data.get("response", [])
+            if groups:
+                return groups[0].get("screen_name")
+    except Exception:
+        pass
+    return None
+
+
 def _source_dict(s: VkSource) -> dict:
    return {
        "id":            str(s.id),
        "group_id":      s.group_id,
        "group_name":    s.group_name,
+        "screen_name":   s.screen_name,
        "enabled":       s.enabled,
        "strict_filter": s.strict_filter,
        "last_run":      s.last_run.isoformat() if s.last_run else None,
@@ -58,7 +78,8 @@ async def add_source(data: SourceIn, db: AsyncSession = Depends(get_db), _: str
    exists = (await db.execute(select(VkSource).where(VkSource.group_id == group_id))).scalar_one_or_none()
    if exists:
        raise HTTPException(status_code=409, detail="Источник с таким group_id уже есть")
-    source = VkSource(group_id=group_id, group_name=data.group_name, enabled=data.enabled, strict_filter=data.strict_filter)
+    screen_name = await _fetch_screen_name(group_id)
+    source = VkSource(group_id=group_id, group_name=data.group_name, screen_name=screen_name, enabled=data.enabled, strict_filter=data.strict_filter)
    db.add(source)
    await db.commit()
    await db.refresh(source)
@@ -74,6 +95,8 @@ async def update_source(source_id: str, data: SourceIn, db: AsyncSession = Depen
    source.group_name    = data.group_name
    source.enabled       = data.enabled
    source.strict_filter = data.strict_filter
+    if not source.screen_name:
+        source.screen_name = await _fetch_screen_name(source.group_id)
    await db.commit()
    await db.refresh(source)
    return _source_dict(source)
--- a/api/vk_parser.py
+++ b/api/vk_parser.py
@@ -162,7 +162,8 @@ async def _upload_from_url(url: str, db: AsyncSession | None = None,
        mc = _minio_client()
        _ensure_bucket(mc)
        mc.put_object(MINIO_BUCKET, object_name, io.BytesIO(data), len(data), content_type=ct)
-        public_url = f"{MINIO_PUBLIC_URL}/{MINIO_BUCKET}/{object_name}"
+        # Системный nginx проксирует /media/ → MinIO bucket, поэтому bucket не дублируем в URL
+        public_url = f"{MINIO_PUBLIC_URL}/{object_name}"
        if db is not None:
            from bd.models import Media
            db.add(Media(
@@ -302,7 +303,7 @@ async def _download_vk_video(owner_id: int, video_id: int,
        mc = _minio_client()
        _ensure_bucket(mc)
        mc.put_object(MINIO_BUCKET, object_name, io.BytesIO(data), len(data), content_type="video/mp4")
-        public_url = f"{MINIO_PUBLIC_URL}/{MINIO_BUCKET}/{object_name}"
+        public_url = f"{MINIO_PUBLIC_URL}/{object_name}"
        print(f"[VK] Видео загружено в MinIO: {object_name} ({total/1024/1024:.1f}MB)")
        if db is not None:
            from bd.models import Media
@@ -360,8 +361,8 @@ async def _slug_exists(slug: str, db: AsyncSession) -> bool:


 async def _vk_post_exists(owner_id: int, post_id: int, db: AsyncSession) -> bool:
-    source_url = f"https://vk.com/wall{owner_id}_{post_id}"
-    r = await db.execute(select(Article.id).where(Article.source_url == source_url))
+    pattern = f"%wall{owner_id}_{post_id}%"
+    r = await db.execute(select(Article.id).where(Article.source_url.like(pattern)))
    return r.scalar_one_or_none() is not None


@@ -387,7 +388,7 @@ async def _get_or_create_tag(name: str, db: AsyncSession) -> Tag:

 # ── Post processing ───────────────────────────────────────────────────────────

-async def _process_post(post: dict, category_name: str, db: AsyncSession, strict_filter: bool = True) -> bool:
+async def _process_post(post: dict, category_name: str, db: AsyncSession, strict_filter: bool = True, screen_name: str | None = None) -> bool:
    owner_id = post["owner_id"]
    post_id  = post["id"]
    vk_slug  = _vk_slug(owner_id, post_id)
@@ -561,7 +562,10 @@ async def _process_post(post: dict, category_name: str, db: AsyncSession, strict
    created_at   = datetime.utcfromtimestamp(ts)
    published_at = created_at if status == ArticleStatus.published else None

-    source_url = f"https://vk.com/wall{owner_id}_{post_id}"
+    if screen_name:
+        source_url = f"https://vk.com/{screen_name}?w=wall{owner_id}_{post_id}"
+    else:
+        source_url = f"https://vk.com/wall{owner_id}_{post_id}"

    article = Article(
        id=article_id,
@@ -672,7 +676,7 @@ async def run_import() -> dict:
                batch_new = 0
                for post in posts:
                    try:
-                        ok = await _process_post(post, source.group_name, db, source.strict_filter)
+                        ok = await _process_post(post, source.group_name, db, source.strict_filter, screen_name=source.screen_name)
                        if ok:
                            imported += 1; grp_new += 1; batch_new += 1
                        else:
--- a/docker-compose.external.yml
+++ b/docker-compose.external.yml
@@ -47,7 +47,7 @@ services:
      api:
        condition: service_healthy
    healthcheck:
-      test: ["CMD-SHELL", "wget -qO- http://localhost/ || exit 1"]
+      test: ["CMD-SHELL", "wget -qO- http://127.0.0.1/ || exit 1"]
      interval: 10s
      timeout: 5s
      retries: 5
--- a/web-react/nginx.conf
+++ b/web-react/nginx.conf
@@ -23,7 +23,6 @@ server {
        proxy_pass         http://api:8000/;
        proxy_set_header   Host $host;
        proxy_set_header   X-Real-IP $remote_addr;
-        # Отключаем буферизацию для быстрого ответа
        proxy_buffering    off;
    }

--- a/web-react/src/api/index.js
+++ b/web-react/src/api/index.js
@@ -114,3 +114,6 @@ export const adminVkImportHistory = (token, sourceId, sinceDays = 365) =>

 export const adminSystemStatus = token =>
  req('GET', 'admin/system/status', token)
+
+export const adminRepairMediaUrls = token =>
+  req('POST', 'admin/system/repair-media-urls', token)
--- a/web-react/src/pages/admin/SystemStatus.jsx
+++ b/web-react/src/pages/admin/SystemStatus.jsx
@@ -1,6 +1,6 @@
 import React, { useState, useEffect, useCallback } from 'react'
 import { useAuth } from '../../hooks/useAuth'
-import { adminSystemStatus } from '../../api/index'
+import { adminSystemStatus, adminRepairMediaUrls } from '../../api/index'
 import AdminLayout from '../../components/AdminLayout'
 import Loader from '../../components/Loader'

@@ -90,6 +90,7 @@ export default function SystemStatus() {
  const [data, setData] = useState(null)
  const [loading, setLoading] = useState(true)
  const [lastUpdated, setLastUpdated] = useState(null)
+  const [repairState, setRepairState] = useState(null) // null | 'running' | result-object

  const load = useCallback(async () => {
    setLoading(true)
@@ -106,6 +107,17 @@ export default function SystemStatus() {

  useEffect(() => { load() }, [load])

+  const runRepair = async () => {
+    if (!window.confirm('Обновить все MinIO URL в базе данных?\nЭто перезапишет cover_url и img-теги в контенте статей.')) return
+    setRepairState('running')
+    try {
+      const result = await adminRepairMediaUrls(token)
+      setRepairState(result)
+    } catch (e) {
+      setRepairState({ error: String(e) })
+    }
+  }
+
  const allOk = data && Object.values(data).every(s => s.ok)

  return (
@@ -127,11 +139,34 @@ export default function SystemStatus() {
            </span>
          )}
        </div>
-        <button className="btn btn-ghost btn-sm" onClick={load} disabled={loading}>
-          {loading ? 'Проверяю...' : '🔄 Обновить'}
-        </button>
+        <div style={{ display: 'flex', gap: '0.5rem', flexWrap: 'wrap' }}>
+          <button className="btn btn-ghost btn-sm" onClick={load} disabled={loading}>
+            {loading ? 'Проверяю...' : '🔄 Обновить'}
+          </button>
+          <button
+            className="btn btn-sm"
+            style={{ background: 'var(--c-warning, #f59e0b)', color: '#fff', border: 'none' }}
+            onClick={runRepair}
+            disabled={repairState === 'running'}
+          >
+            {repairState === 'running' ? 'Исправляю...' : '🔧 Починить URL медиа'}
+          </button>
+        </div>
      </div>

+      {repairState && repairState !== 'running' && (
+        <div style={{
+          background: repairState.error ? 'var(--c-danger-bg, #fee2e2)' : 'var(--c-success-bg, #d1fae5)',
+          color: repairState.error ? 'var(--c-danger, #991b1b)' : 'var(--c-success, #065f46)',
+          borderRadius: 8, padding: '0.75rem 1rem', marginBottom: '1rem', fontSize: '0.875rem',
+        }}>
+          {repairState.error
+            ? `Ошибка: ${repairState.error}`
+            : `✅ Готово. Новый base: ${repairState.new_base} | cover_url: ${repairState.cover_url_fixed} | контент: ${repairState.content_fixed} | media: ${repairState.media_fixed}`
+          }
+        </div>
+      )}
+
      {loading && !data ? <Loader /> : (
        <div style={{ display: 'grid', gap: '1rem', gridTemplateColumns: 'repeat(auto-fill, minmax(340px, 1fr))' }}>
          <ServiceCard title="PostgreSQL" icon="🐘" data={data?.postgres} />
Author	SHA1	Message	Date
jze9	615c819e97	fix: корректные source_url для VK постов (screen_name формат) - VkSource: добавлено поле screen_name, заполняется из VK API при добавлении источника - vk_parser: source_url теперь в формате vk.com/{screen_name}?w=wall{owner}_{post} вместо устаревшего vk.com/wall{owner}_{post} - _vk_post_exists: проверка через LIKE чтобы находить оба формата URL - admin_vk: автоматически получает screen_name при создании/обновлении источника - admin_system: исправлен regex в repair-media-urls — только /news-media/ URL - Alembic миграция: b2c3d4e5f6a7 добавляет колонку screen_name в vk_sources Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-20 19:32:28 +05:00
jze9	9e277de65d	fix: исправить URL медиа — nginx /media/ проксирует без имени бакета - vk_parser: убрать bucket из MINIO_PUBLIC_URL (nginx добавляет /news-media/ сам) - admin_system: добавить endpoint POST /repair-media-urls для обновления старых URL в БД - SystemStatus: кнопка «Починить URL медиа» запускает repair endpoint - .env.example: заменить реальные адреса/пароли на плейсхолдеры Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-20 16:44:36 +05:00
jze9	00acc59cc4	Revert "nginx: добавлен конфиг для news.cloud-copp74.ru, порт web→8080" This reverts commit `764c446efd`.	2026-05-19 19:08:13 +05:00
jze9	764c446efd	nginx: добавлен конфиг для news.cloud-copp74.ru, порт web→8080 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-19 18:39:18 +05:00
jze9	6de9464401	fix: healthcheck web container uses 127.0.0.1 instead of localhost Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-19 17:53:55 +05:00